2023-10-20T09:00:14.063 INFO:root:teuthology version: 0.0.1.dev201+g202b180 2023-10-20T09:00:14.065 DEBUG:teuthology.run:Teuthology command: teuthology --description rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} --name rraja-2023-10-20_08:52:26-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi --archive /home/teuthworker/archive/rraja-2023-10-20_08:52:26-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7432983 --owner scheduled_rraja@teuthology --verbose -- /home/teuthworker/archive/rraja-2023-10-20_08:52:26-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7432983/orig.config.yaml 2023-10-20T09:00:14.099 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-20T09:00:14.181 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rraja-2023-10-20_08:52:26-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7432983 branch: ajarr-wip-extra-debug-62994-oct-19 description: rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} email: rraja@redhat.com first_in_suite: false job_id: '7432983' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: rraja-2023-10-20_08:52:26-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: ajarr-wip-extra-debug-62994-oct-19 ceph: conf: client: rbd default data pool: datapool rbd default features: 1 global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 debug rbd: 20 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: ajarr-wip-extra-debug-62994-oct-19 sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab owner: scheduled_rraja@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 9138 sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab sleep_before_teardown: 0 suite: rbd:cli suite_branch: ajarr-wip-extra-debug-62994-oct-19 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab targets: smithi150.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMhNZKSaorwOILbIWwC8yum4ac4p9BDJPPbLNv44G7t2YNONzis8NtcZXgBhQBBlI9Fi9CxItahxOgAuWR2Gut0= tasks: - install: null - ceph: null - exec: client.0: - sudo ceph osd pool create datapool 4 - rbd pool init datapool - install: extra_system_packages: - fio - workunit: clients: client.0: - rbd/rbd_support_module_recovery.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 202b180cb047e798fb131047314a862593f45403 timestamp: 2023-10-20_08:52:26 tube: smithi user: rraja verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-20T09:00:14.181 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa; will attempt to use it 2023-10-20T09:00:14.182 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa/tasks 2023-10-20T09:00:14.182 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-20T09:00:14.184 INFO:teuthology.task.internal:Checking packages... 2023-10-20T09:00:14.208 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab' 2023-10-20T09:00:14.209 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-20T09:00:14.209 INFO:teuthology.packaging:ref: None 2023-10-20T09:00:14.210 INFO:teuthology.packaging:tag: None 2023-10-20T09:00:14.210 INFO:teuthology.packaging:branch: ajarr-wip-extra-debug-62994-oct-19 2023-10-20T09:00:14.210 INFO:teuthology.packaging:sha1: 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-20T09:00:14.211 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=ajarr-wip-extra-debug-62994-oct-19 2023-10-20T09:00:14.431 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6744-g7f103c7c-1jammy 2023-10-20T09:00:14.432 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-20T09:00:14.441 INFO:teuthology.task.internal:no buildpackages task found 2023-10-20T09:00:14.442 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-20T09:00:14.489 INFO:teuthology.task.internal:Saving configuration 2023-10-20T09:00:14.502 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-20T09:00:14.520 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-20T09:00:14.545 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rraja-2023-10-20_08:52:26-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7432983', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-20 08:54:30.421475', 'locked_by': 'scheduled_rraja@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBPyxHtKfeBbsNsM5WLaZ0qYvFzraysRNB2A5UU6Jpm5'} 2023-10-20T09:00:14.546 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-20T09:00:14.553 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-20T09:00:14.554 INFO:teuthology.run_tasks:Running task console_log... 2023-10-20T09:00:14.624 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd7600b0a60>, signals=[15]) 2023-10-20T09:00:14.625 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-20T09:00:14.633 INFO:teuthology.task.internal:Opening connections... 2023-10-20T09:00:14.633 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi150.front.sepia.ceph.com 2023-10-20T09:00:14.635 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T09:00:14.713 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-20T09:00:14.723 DEBUG:teuthology.orchestra.run.smithi150:> uname -m 2023-10-20T09:00:14.830 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2023-10-20T09:00:14.831 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release 2023-10-20T09:00:14.877 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-20T09:00:14.877 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="Ubuntu" 2023-10-20T09:00:14.877 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="22.04" 2023-10-20T09:00:14.878 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-20T09:00:14.878 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_CODENAME=jammy 2023-10-20T09:00:14.878 INFO:teuthology.orchestra.run.smithi150.stdout:ID=ubuntu 2023-10-20T09:00:14.878 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE=debian 2023-10-20T09:00:14.878 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-20T09:00:14.879 INFO:teuthology.orchestra.run.smithi150.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-20T09:00:14.879 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-20T09:00:14.879 INFO:teuthology.orchestra.run.smithi150.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-20T09:00:14.879 INFO:teuthology.orchestra.run.smithi150.stdout:UBUNTU_CODENAME=jammy 2023-10-20T09:00:14.880 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2023-10-20T09:00:14.908 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-20T09:00:14.917 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-20T09:00:14.958 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-20T09:00:14.959 DEBUG:teuthology.orchestra.run.smithi150:> test '!' -e /home/ubuntu/cephtest 2023-10-20T09:00:14.964 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-20T09:00:14.991 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-20T09:00:14.992 DEBUG:teuthology.orchestra.run.smithi150:> test -z $(ls -A /var/lib/ceph) 2023-10-20T09:00:15.013 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-20T09:00:15.058 INFO:teuthology.run_tasks:Running task kernel... 2023-10-20T09:00:15.089 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-20T09:00:15.090 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-20T09:00:15.091 DEBUG:teuthology.orchestra.run.smithi150:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-20T09:00:15.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T09:00:15.096 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2023-10-20T09:00:15.142 INFO:teuthology.orchestra.run.smithi150.stdout:5.15.0-56-generic 2023-10-20T09:00:15.142 INFO:teuthology.task.kernel:Running kernel on smithi150: 5.15.0-56-generic 2023-10-20T09:00:15.143 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-get clean 2023-10-20T09:00:15.275 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-get update 2023-10-20T09:00:15.408 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-20T09:00:15.441 INFO:teuthology.orchestra.run.smithi150.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-10-20T09:00:15.513 INFO:teuthology.orchestra.run.smithi150.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-10-20T09:00:15.517 INFO:teuthology.orchestra.run.smithi150.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-10-20T09:00:15.633 INFO:teuthology.orchestra.run.smithi150.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,103 kB] 2023-10-20T09:00:15.719 INFO:teuthology.orchestra.run.smithi150.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [515 kB] 2023-10-20T09:00:15.725 INFO:teuthology.orchestra.run.smithi150.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [239 kB] 2023-10-20T09:00:15.728 INFO:teuthology.orchestra.run.smithi150.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.0 kB] 2023-10-20T09:00:15.729 INFO:teuthology.orchestra.run.smithi150.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,036 kB] 2023-10-20T09:00:15.745 INFO:teuthology.orchestra.run.smithi150.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.4 kB] 2023-10-20T09:00:15.745 INFO:teuthology.orchestra.run.smithi150.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [167 kB] 2023-10-20T09:00:15.747 INFO:teuthology.orchestra.run.smithi150.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [536 B] 2023-10-20T09:00:15.749 INFO:teuthology.orchestra.run.smithi150.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [995 kB] 2023-10-20T09:00:15.759 INFO:teuthology.orchestra.run.smithi150.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [660 kB] 2023-10-20T09:00:15.764 INFO:teuthology.orchestra.run.smithi150.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [218 kB] 2023-10-20T09:00:15.790 INFO:teuthology.orchestra.run.smithi150.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.0 kB] 2023-10-20T09:00:15.792 INFO:teuthology.orchestra.run.smithi150.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-10-20T09:00:15.792 INFO:teuthology.orchestra.run.smithi150.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-10-20T09:00:15.792 INFO:teuthology.orchestra.run.smithi150.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-10-20T09:00:15.793 INFO:teuthology.orchestra.run.smithi150.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-10-20T09:00:15.793 INFO:teuthology.orchestra.run.smithi150.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-10-20T09:00:15.793 INFO:teuthology.orchestra.run.smithi150.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-10-20T09:00:15.794 INFO:teuthology.orchestra.run.smithi150.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-10-20T09:00:15.794 INFO:teuthology.orchestra.run.smithi150.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-10-20T09:00:15.810 INFO:teuthology.orchestra.run.smithi150.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.9 kB] 2023-10-20T09:00:15.830 INFO:teuthology.orchestra.run.smithi150.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-10-20T09:00:15.830 INFO:teuthology.orchestra.run.smithi150.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-10-20T09:00:15.831 INFO:teuthology.orchestra.run.smithi150.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-10-20T09:00:15.939 INFO:teuthology.orchestra.run.smithi150.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [353 kB] 2023-10-20T09:00:16.201 INFO:teuthology.orchestra.run.smithi150.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [896 kB] 2023-10-20T09:00:16.347 INFO:teuthology.orchestra.run.smithi150.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [180 kB] 2023-10-20T09:00:16.367 INFO:teuthology.orchestra.run.smithi150.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-10-20T09:00:16.368 INFO:teuthology.orchestra.run.smithi150.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.0 kB] 2023-10-20T09:00:16.370 INFO:teuthology.orchestra.run.smithi150.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,016 kB] 2023-10-20T09:00:16.436 INFO:teuthology.orchestra.run.smithi150.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [164 kB] 2023-10-20T09:00:16.446 INFO:teuthology.orchestra.run.smithi150.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [536 B] 2023-10-20T09:00:16.446 INFO:teuthology.orchestra.run.smithi150.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [562 kB] 2023-10-20T09:00:16.471 INFO:teuthology.orchestra.run.smithi150.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [793 kB] 2023-10-20T09:00:16.660 INFO:teuthology.orchestra.run.smithi150.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [145 kB] 2023-10-20T09:00:17.042 INFO:teuthology.orchestra.run.smithi150.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-10-20T09:00:17.049 INFO:teuthology.orchestra.run.smithi150.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-10-20T09:00:17.049 INFO:teuthology.orchestra.run.smithi150.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-10-20T09:00:17.064 INFO:teuthology.orchestra.run.smithi150.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-10-20T09:00:17.068 INFO:teuthology.orchestra.run.smithi150.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-10-20T09:00:21.602 INFO:teuthology.orchestra.run.smithi150.stdout:Fetched 9,805 kB in 2s (5,659 kB/s) 2023-10-20T09:00:22.876 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2023-10-20T09:00:22.893 DEBUG:teuthology.orchestra.run.smithi150:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-20T09:00:22.953 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2023-10-20T09:00:23.147 INFO:teuthology.orchestra.run.smithi150.stdout:Building dependency tree... 2023-10-20T09:00:23.148 INFO:teuthology.orchestra.run.smithi150.stdout:Reading state information... 2023-10-20T09:00:23.301 INFO:teuthology.orchestra.run.smithi150.stdout:The following packages were automatically installed and are no longer required: 2023-10-20T09:00:23.301 INFO:teuthology.orchestra.run.smithi150.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-20T09:00:23.302 INFO:teuthology.orchestra.run.smithi150.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-20T09:00:23.302 INFO:teuthology.orchestra.run.smithi150.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-20T09:00:23.303 INFO:teuthology.orchestra.run.smithi150.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-20T09:00:23.303 INFO:teuthology.orchestra.run.smithi150.stdout: python2.7-minimal 2023-10-20T09:00:23.303 INFO:teuthology.orchestra.run.smithi150.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-20T09:00:23.303 INFO:teuthology.orchestra.run.smithi150.stdout:The following additional packages will be installed: 2023-10-20T09:00:23.304 INFO:teuthology.orchestra.run.smithi150.stdout: linux-generic linux-headers-5.15.0-87 linux-headers-5.15.0-87-generic 2023-10-20T09:00:23.304 INFO:teuthology.orchestra.run.smithi150.stdout: linux-headers-generic linux-image-5.15.0-87-generic 2023-10-20T09:00:23.304 INFO:teuthology.orchestra.run.smithi150.stdout: linux-modules-5.15.0-87-generic linux-modules-extra-5.15.0-87-generic 2023-10-20T09:00:23.305 INFO:teuthology.orchestra.run.smithi150.stdout:Suggested packages: 2023-10-20T09:00:23.305 INFO:teuthology.orchestra.run.smithi150.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-10-20T09:00:23.305 INFO:teuthology.orchestra.run.smithi150.stdout:Recommended packages: 2023-10-20T09:00:23.305 INFO:teuthology.orchestra.run.smithi150.stdout: thermald 2023-10-20T09:00:23.355 INFO:teuthology.orchestra.run.smithi150.stdout:The following NEW packages will be installed: 2023-10-20T09:00:23.356 INFO:teuthology.orchestra.run.smithi150.stdout: linux-headers-5.15.0-87 linux-headers-5.15.0-87-generic 2023-10-20T09:00:23.356 INFO:teuthology.orchestra.run.smithi150.stdout: linux-image-5.15.0-87-generic linux-modules-5.15.0-87-generic 2023-10-20T09:00:23.356 INFO:teuthology.orchestra.run.smithi150.stdout: linux-modules-extra-5.15.0-87-generic 2023-10-20T09:00:23.357 INFO:teuthology.orchestra.run.smithi150.stdout:The following packages will be upgraded: 2023-10-20T09:00:23.357 INFO:teuthology.orchestra.run.smithi150.stdout: linux-generic linux-headers-generic linux-image-generic 2023-10-20T09:00:26.166 INFO:teuthology.orchestra.run.smithi150.stdout:3 upgraded, 5 newly installed, 0 to remove and 269 not upgraded. 2023-10-20T09:00:26.166 INFO:teuthology.orchestra.run.smithi150.stdout:Need to get 113 MB of archives. 2023-10-20T09:00:26.166 INFO:teuthology.orchestra.run.smithi150.stdout:After this operation, 582 MB of additional disk space will be used. 2023-10-20T09:00:26.167 INFO:teuthology.orchestra.run.smithi150.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-87-generic amd64 5.15.0-87.97 [22.4 MB] 2023-10-20T09:00:26.626 INFO:teuthology.orchestra.run.smithi150.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-87-generic amd64 5.15.0-87.97 [11.5 MB] 2023-10-20T09:00:26.744 INFO:teuthology.orchestra.run.smithi150.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-87-generic amd64 5.15.0-87.97 [64.0 MB] 2023-10-20T09:00:27.675 INFO:teuthology.orchestra.run.smithi150.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.87.84 [1,698 B] 2023-10-20T09:00:27.676 INFO:teuthology.orchestra.run.smithi150.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.87.84 [2,482 B] 2023-10-20T09:00:27.676 INFO:teuthology.orchestra.run.smithi150.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-87 all 5.15.0-87.97 [12.3 MB] 2023-10-20T09:00:27.802 INFO:teuthology.orchestra.run.smithi150.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-87-generic amd64 5.15.0-87.97 [2,888 kB] 2023-10-20T09:00:27.845 INFO:teuthology.orchestra.run.smithi150.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.87.84 [2,322 B] 2023-10-20T09:00:28.270 INFO:teuthology.orchestra.run.smithi150.stdout:Fetched 113 MB in 4s (25.2 MB/s) 2023-10-20T09:00:28.445 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package linux-modules-5.15.0-87-generic. 2023-10-20T09:00:30.527 INFO:teuthology.orchestra.run.smithi150.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-10-20T09:00:30.531 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../0-linux-modules-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-20T09:00:30.626 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-modules-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T09:00:32.827 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package linux-image-5.15.0-87-generic. 2023-10-20T09:00:32.849 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../1-linux-image-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-20T09:00:32.947 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T09:00:33.439 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-87-generic. 2023-10-20T09:00:33.459 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-20T09:00:33.497 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-modules-extra-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T09:00:40.517 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../3-linux-generic_5.15.0.87.84_amd64.deb ... 2023-10-20T09:00:40.634 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-20T09:00:40.924 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.87.84_amd64.deb ... 2023-10-20T09:00:41.054 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-image-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-20T09:00:41.365 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package linux-headers-5.15.0-87. 2023-10-20T09:00:41.387 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../5-linux-headers-5.15.0-87_5.15.0-87.97_all.deb ... 2023-10-20T09:00:41.432 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-headers-5.15.0-87 (5.15.0-87.97) ... 2023-10-20T09:00:47.619 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package linux-headers-5.15.0-87-generic. 2023-10-20T09:00:47.656 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../6-linux-headers-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-20T09:00:47.686 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-headers-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T09:00:50.317 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.87.84_amd64.deb ... 2023-10-20T09:00:50.440 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-headers-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-20T09:00:50.807 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-headers-5.15.0-87 (5.15.0-87.97) ... 2023-10-20T09:00:50.928 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-headers-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T09:00:51.563 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-headers-generic (5.15.0.87.84) ... 2023-10-20T09:00:51.690 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-modules-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T09:00:53.101 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T09:00:54.453 INFO:teuthology.orchestra.run.smithi150.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-10-20T09:00:54.454 INFO:teuthology.orchestra.run.smithi150.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-10-20T09:00:54.454 INFO:teuthology.orchestra.run.smithi150.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-87-generic 2023-10-20T09:00:54.454 INFO:teuthology.orchestra.run.smithi150.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-87-generic 2023-10-20T09:00:54.595 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-modules-extra-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T09:00:56.004 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-image-generic (5.15.0.87.84) ... 2023-10-20T09:00:56.105 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-generic (5.15.0.87.84) ... 2023-10-20T09:00:56.215 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T09:00:56.285 INFO:teuthology.orchestra.run.smithi150.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-10-20T09:00:56.286 INFO:teuthology.orchestra.run.smithi150.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-87-generic 2023-10-20T09:01:10.600 INFO:teuthology.orchestra.run.smithi150.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-10-20T09:01:10.601 INFO:teuthology.orchestra.run.smithi150.stdout:Sourcing file `/etc/default/grub' 2023-10-20T09:01:10.628 INFO:teuthology.orchestra.run.smithi150.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-20T09:01:10.656 INFO:teuthology.orchestra.run.smithi150.stdout:Generating grub configuration file ... 2023-10-20T09:01:10.930 INFO:teuthology.orchestra.run.smithi150.stdout:Found linux image: /boot/vmlinuz-5.15.0-87-generic 2023-10-20T09:01:10.955 INFO:teuthology.orchestra.run.smithi150.stdout:Found initrd image: /boot/initrd.img-5.15.0-87-generic 2023-10-20T09:01:11.300 INFO:teuthology.orchestra.run.smithi150.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-20T09:01:11.305 INFO:teuthology.orchestra.run.smithi150.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-20T09:01:11.372 INFO:teuthology.orchestra.run.smithi150.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-20T09:01:11.378 INFO:teuthology.orchestra.run.smithi150.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-20T09:01:11.521 INFO:teuthology.orchestra.run.smithi150.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-20T09:01:11.522 INFO:teuthology.orchestra.run.smithi150.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-10-20T09:01:11.522 INFO:teuthology.orchestra.run.smithi150.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-20T09:01:11.577 INFO:teuthology.orchestra.run.smithi150.stdout:done 2023-10-20T09:01:12.046 DEBUG:teuthology.orchestra.run.smithi150:> dpkg -s linux-image-generic 2023-10-20T09:01:12.069 INFO:teuthology.orchestra.run.smithi150.stdout:Package: linux-image-generic 2023-10-20T09:01:12.070 INFO:teuthology.orchestra.run.smithi150.stdout:Status: install ok installed 2023-10-20T09:01:12.070 INFO:teuthology.orchestra.run.smithi150.stdout:Priority: optional 2023-10-20T09:01:12.070 INFO:teuthology.orchestra.run.smithi150.stdout:Section: kernel 2023-10-20T09:01:12.070 INFO:teuthology.orchestra.run.smithi150.stdout:Installed-Size: 20 2023-10-20T09:01:12.071 INFO:teuthology.orchestra.run.smithi150.stdout:Maintainer: Ubuntu Kernel Team 2023-10-20T09:01:12.071 INFO:teuthology.orchestra.run.smithi150.stdout:Architecture: amd64 2023-10-20T09:01:12.071 INFO:teuthology.orchestra.run.smithi150.stdout:Source: linux-meta 2023-10-20T09:01:12.071 INFO:teuthology.orchestra.run.smithi150.stdout:Version: 5.15.0.87.84 2023-10-20T09:01:12.071 INFO:teuthology.orchestra.run.smithi150.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-87), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-20T09:01:12.072 INFO:teuthology.orchestra.run.smithi150.stdout:Depends: linux-image-5.15.0-87-generic, linux-modules-extra-5.15.0-87-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-20T09:01:12.072 INFO:teuthology.orchestra.run.smithi150.stdout:Recommends: thermald 2023-10-20T09:01:12.072 INFO:teuthology.orchestra.run.smithi150.stdout:Description: Generic Linux kernel image 2023-10-20T09:01:12.073 INFO:teuthology.orchestra.run.smithi150.stdout: This package will always depend on the latest generic kernel image 2023-10-20T09:01:12.073 INFO:teuthology.orchestra.run.smithi150.stdout: available. 2023-10-20T09:01:12.073 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-87-generic 2023-10-20T09:01:12.074 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-20T09:01:12.074 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-20T09:01:12.074 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-20T09:01:12.074 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi150.front.sepia.ceph.com, path=None, version=distro) 2023-10-20T09:01:12.075 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-get clean 2023-10-20T09:01:12.223 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-get update 2023-10-20T09:01:12.468 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-20T09:01:12.468 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-20T09:01:12.536 INFO:teuthology.orchestra.run.smithi150.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-10-20T09:01:12.890 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-20T09:01:13.052 INFO:teuthology.orchestra.run.smithi150.stdout:Fetched 119 kB in 1s (173 kB/s) 2023-10-20T09:01:13.938 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2023-10-20T09:01:13.955 DEBUG:teuthology.orchestra.run.smithi150:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-20T09:01:14.007 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2023-10-20T09:01:14.171 INFO:teuthology.orchestra.run.smithi150.stdout:Building dependency tree... 2023-10-20T09:01:14.172 INFO:teuthology.orchestra.run.smithi150.stdout:Reading state information... 2023-10-20T09:01:14.313 INFO:teuthology.orchestra.run.smithi150.stdout:linux-image-generic is already the newest version (5.15.0.87.84). 2023-10-20T09:01:14.314 INFO:teuthology.orchestra.run.smithi150.stdout:The following packages were automatically installed and are no longer required: 2023-10-20T09:01:14.314 INFO:teuthology.orchestra.run.smithi150.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-20T09:01:14.314 INFO:teuthology.orchestra.run.smithi150.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-20T09:01:14.315 INFO:teuthology.orchestra.run.smithi150.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-20T09:01:14.315 INFO:teuthology.orchestra.run.smithi150.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-20T09:01:14.315 INFO:teuthology.orchestra.run.smithi150.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-20T09:01:14.316 INFO:teuthology.orchestra.run.smithi150.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-20T09:01:14.351 INFO:teuthology.orchestra.run.smithi150.stdout:0 upgraded, 0 newly installed, 0 to remove and 269 not upgraded. 2023-10-20T09:01:14.353 DEBUG:teuthology.orchestra.run.smithi150:> dpkg -s linux-image-generic 2023-10-20T09:01:14.412 INFO:teuthology.orchestra.run.smithi150.stdout:Package: linux-image-generic 2023-10-20T09:01:14.413 INFO:teuthology.orchestra.run.smithi150.stdout:Status: install ok installed 2023-10-20T09:01:14.413 INFO:teuthology.orchestra.run.smithi150.stdout:Priority: optional 2023-10-20T09:01:14.413 INFO:teuthology.orchestra.run.smithi150.stdout:Section: kernel 2023-10-20T09:01:14.413 INFO:teuthology.orchestra.run.smithi150.stdout:Installed-Size: 20 2023-10-20T09:01:14.414 INFO:teuthology.orchestra.run.smithi150.stdout:Maintainer: Ubuntu Kernel Team 2023-10-20T09:01:14.414 INFO:teuthology.orchestra.run.smithi150.stdout:Architecture: amd64 2023-10-20T09:01:14.414 INFO:teuthology.orchestra.run.smithi150.stdout:Source: linux-meta 2023-10-20T09:01:14.414 INFO:teuthology.orchestra.run.smithi150.stdout:Version: 5.15.0.87.84 2023-10-20T09:01:14.414 INFO:teuthology.orchestra.run.smithi150.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-87), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-20T09:01:14.415 INFO:teuthology.orchestra.run.smithi150.stdout:Depends: linux-image-5.15.0-87-generic, linux-modules-extra-5.15.0-87-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-20T09:01:14.415 INFO:teuthology.orchestra.run.smithi150.stdout:Recommends: thermald 2023-10-20T09:01:14.415 INFO:teuthology.orchestra.run.smithi150.stdout:Description: Generic Linux kernel image 2023-10-20T09:01:14.415 INFO:teuthology.orchestra.run.smithi150.stdout: This package will always depend on the latest generic kernel image 2023-10-20T09:01:14.415 INFO:teuthology.orchestra.run.smithi150.stdout: available. 2023-10-20T09:01:14.416 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2023-10-20T09:01:14.461 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.iodmsvK0HH 2023-10-20T09:01:14.461 DEBUG:teuthology.orchestra.run.smithi150:> sudo cp /boot/grub/grub.cfg /tmp/tmp.iodmsvK0HH 2023-10-20T09:01:14.516 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.iodmsvK0HH 2023-10-20T09:01:14.634 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.iodmsvK0HH is 10KB 2023-10-20T09:01:14.647 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.iodmsvK0HH 2023-10-20T09:01:14.654 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-10-20T09:01:14.708 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-20T09:01:14.708 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-10-20T09:01:14.709 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-10-20T09:01:14.782 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-87-generic 2023-10-20T09:01:14.782 DEBUG:teuthology.orchestra.run.smithi150:> sudo update-grub 2023-10-20T09:01:16.033 INFO:teuthology.orchestra.run.smithi150.stderr:Sourcing file `/etc/default/grub' 2023-10-20T09:01:16.034 INFO:teuthology.orchestra.run.smithi150.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-20T09:01:16.037 INFO:teuthology.orchestra.run.smithi150.stderr:Generating grub configuration file ... 2023-10-20T09:01:16.230 INFO:teuthology.orchestra.run.smithi150.stderr:Found linux image: /boot/vmlinuz-5.15.0-87-generic 2023-10-20T09:01:16.236 INFO:teuthology.orchestra.run.smithi150.stderr:Found initrd image: /boot/initrd.img-5.15.0-87-generic 2023-10-20T09:01:16.516 INFO:teuthology.orchestra.run.smithi150.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-20T09:01:16.522 INFO:teuthology.orchestra.run.smithi150.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-20T09:01:16.562 INFO:teuthology.orchestra.run.smithi150.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-20T09:01:16.567 INFO:teuthology.orchestra.run.smithi150.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-20T09:01:16.686 INFO:teuthology.orchestra.run.smithi150.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-20T09:01:16.686 INFO:teuthology.orchestra.run.smithi150.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-20T09:01:16.686 INFO:teuthology.orchestra.run.smithi150.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-20T09:01:16.700 INFO:teuthology.orchestra.run.smithi150.stderr:done 2023-10-20T09:01:16.702 DEBUG:teuthology.orchestra.run.smithi150:> sudo shutdown -r now 2023-10-20T09:01:46.731 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-20T09:01:46.732 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-10-20T09:01:46.732 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T09:02:05.179 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-10-20T09:02:14.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-10-20T09:02:14.188 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T09:02:17.242 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-10-20T09:02:29.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-10-20T09:02:29.255 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T09:02:47.678 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-10-20T09:03:02.762 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-10-20T09:03:02.763 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T09:03:02.765 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-10-20T09:03:20.782 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-10-20T09:03:20.783 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T09:03:21.040 DEBUG:teuthology.orchestra.run.smithi150:> true 2023-10-20T09:03:21.607 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-10-20T09:03:21.608 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-87-generic"... 2023-10-20T09:03:21.608 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2023-10-20T09:03:21.654 INFO:teuthology.orchestra.run.smithi150.stdout:5.15.0-87-generic 2023-10-20T09:03:21.655 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-87-generic vs 5.15.0-87-generic 2023-10-20T09:03:21.655 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-20T09:03:21.655 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-20T09:03:22.657 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-20T09:03:22.657 DEBUG:teuthology.orchestra.run.smithi150:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-20T09:03:22.737 INFO:teuthology.orchestra.run.smithi150.stdout:ttyS1 2023-10-20T09:03:22.756 DEBUG:teuthology.parallel:result is None 2023-10-20T09:03:22.757 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-20T09:03:22.765 INFO:teuthology.task.internal:Creating test directory... 2023-10-20T09:03:22.765 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-20T09:03:22.772 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-20T09:03:22.800 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-20T09:03:22.837 INFO:teuthology.task.internal:Creating archive directory... 2023-10-20T09:03:22.838 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-20T09:03:22.872 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-20T09:03:22.883 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-20T09:03:22.884 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-20T09:03:22.933 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-20T09:03:22.943 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-20T09:03:22.944 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-20T09:03:22.952 INFO:teuthology.task.internal:Configuring sudo... 2023-10-20T09:03:22.952 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-20T09:03:22.998 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-20T09:03:23.023 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-20T09:03:23.023 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-20T09:03:23.046 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-20T09:03:23.095 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-20T09:03:23.143 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-20T09:03:23.143 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-20T09:03:23.204 DEBUG:teuthology.orchestra.run.smithi150:> sudo service rsyslog restart 2023-10-20T09:03:23.321 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-20T09:03:23.329 INFO:teuthology.task.internal:Starting timer... 2023-10-20T09:03:23.329 INFO:teuthology.run_tasks:Running task pcp... 2023-10-20T09:03:23.370 INFO:teuthology.run_tasks:Running task selinux... 2023-10-20T09:03:23.403 DEBUG:teuthology.task.selinux:Excluding smithi150: OS 'ubuntu' does not support SELinux 2023-10-20T09:03:23.403 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-20T09:03:23.404 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-10-20T09:03:23.404 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-20T09:03:23.404 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-20T09:03:23.436 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-20T09:03:23.437 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-20T09:03:23.454 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-20T09:03:23.456 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi150.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-20T09:11:12.045 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi150.front.sepia.ceph.com')] 2023-10-20T09:11:12.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-10-20T09:11:12.048 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T09:11:12.130 DEBUG:teuthology.orchestra.run.smithi150:> true 2023-10-20T09:11:12.231 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-10-20T09:11:12.231 INFO:teuthology.run_tasks:Running task clock... 2023-10-20T09:11:12.242 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-20T09:11:12.243 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-20T09:11:12.243 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-20T09:11:12.298 INFO:teuthology.orchestra.run.smithi150.stdout:20 Oct 09:11:12 ntpd[28020]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-20T09:11:12.298 INFO:teuthology.orchestra.run.smithi150.stdout:20 Oct 09:11:12 ntpd[28020]: Command line: ntpd -gq 2023-10-20T09:11:12.298 INFO:teuthology.orchestra.run.smithi150.stdout:20 Oct 09:11:12 ntpd[28020]: ---------------------------------------------------- 2023-10-20T09:11:12.299 INFO:teuthology.orchestra.run.smithi150.stdout:20 Oct 09:11:12 ntpd[28020]: ntp-4 is maintained by Network Time Foundation, 2023-10-20T09:11:12.299 INFO:teuthology.orchestra.run.smithi150.stdout:20 Oct 09:11:12 ntpd[28020]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-20T09:11:12.299 INFO:teuthology.orchestra.run.smithi150.stdout:20 Oct 09:11:12 ntpd[28020]: corporation. Support and training for ntp-4 are 2023-10-20T09:11:12.299 INFO:teuthology.orchestra.run.smithi150.stdout:20 Oct 09:11:12 ntpd[28020]: available at https://www.nwtime.org/support 2023-10-20T09:11:12.299 INFO:teuthology.orchestra.run.smithi150.stdout:20 Oct 09:11:12 ntpd[28020]: ---------------------------------------------------- 2023-10-20T09:11:12.300 INFO:teuthology.orchestra.run.smithi150.stdout:20 Oct 09:11:12 ntpd[28020]: proto: precision = 0.044 usec (-24) 2023-10-20T09:11:12.300 INFO:teuthology.orchestra.run.smithi150.stdout:20 Oct 09:11:12 ntpd[28020]: basedate set to 2022-02-04 2023-10-20T09:11:12.301 INFO:teuthology.orchestra.run.smithi150.stdout:20 Oct 09:11:12 ntpd[28020]: gps base set to 2022-02-06 (week 2196) 2023-10-20T09:11:12.301 INFO:teuthology.orchestra.run.smithi150.stdout:20 Oct 09:11:12 ntpd[28020]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-20T09:11:12.301 INFO:teuthology.orchestra.run.smithi150.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-20T09:11:12.302 INFO:teuthology.orchestra.run.smithi150.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-20T09:11:12.303 INFO:teuthology.orchestra.run.smithi150.stdout:20 Oct 09:11:12 ntpd[28020]: restrict ::: KOD does nothing without LIMITED. 2023-10-20T09:11:12.303 INFO:teuthology.orchestra.run.smithi150.stdout:20 Oct 09:11:12 ntpd[28020]: Listen and drop on 0 v6wildcard [::]:123 2023-10-20T09:11:12.303 INFO:teuthology.orchestra.run.smithi150.stdout:20 Oct 09:11:12 ntpd[28020]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-20T09:11:12.303 INFO:teuthology.orchestra.run.smithi150.stdout:20 Oct 09:11:12 ntpd[28020]: Listen normally on 2 lo 127.0.0.1:123 2023-10-20T09:11:12.303 INFO:teuthology.orchestra.run.smithi150.stdout:20 Oct 09:11:12 ntpd[28020]: Listen normally on 3 enp3s0f1 172.21.15.150:123 2023-10-20T09:11:12.304 INFO:teuthology.orchestra.run.smithi150.stdout:20 Oct 09:11:12 ntpd[28020]: Listen normally on 4 lo [::1]:123 2023-10-20T09:11:12.304 INFO:teuthology.orchestra.run.smithi150.stdout:20 Oct 09:11:12 ntpd[28020]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6e35%5]:123 2023-10-20T09:11:12.304 INFO:teuthology.orchestra.run.smithi150.stdout:20 Oct 09:11:12 ntpd[28020]: Listening on routing socket on fd #22 for interface updates 2023-10-20T09:11:14.299 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:14 ntpd[28020]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T09:11:14.299 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:14 ntpd[28020]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T09:11:14.299 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:14 ntpd[28020]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T09:11:14.300 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:14 ntpd[28020]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T09:11:14.300 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:14 ntpd[28020]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T09:11:14.300 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:14 ntpd[28020]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T09:11:14.300 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:14 ntpd[28020]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T09:11:14.300 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:14 ntpd[28020]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T09:11:16.299 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:16 ntpd[28020]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T09:11:16.299 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:16 ntpd[28020]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T09:11:16.299 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:16 ntpd[28020]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T09:11:16.300 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:16 ntpd[28020]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T09:11:16.300 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:16 ntpd[28020]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T09:11:16.300 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:16 ntpd[28020]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T09:11:17.299 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:17 ntpd[28020]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T09:11:17.300 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:17 ntpd[28020]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T09:11:17.300 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:17 ntpd[28020]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T09:11:17.300 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:17 ntpd[28020]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T09:11:17.300 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:17 ntpd[28020]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T09:11:17.300 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:17 ntpd[28020]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T09:11:17.301 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:17 ntpd[28020]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T09:11:17.301 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:17 ntpd[28020]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T09:11:17.301 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:17 ntpd[28020]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T09:11:17.301 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:17 ntpd[28020]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T09:11:17.301 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:17 ntpd[28020]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T09:11:17.302 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:17 ntpd[28020]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T09:11:18.299 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:18 ntpd[28020]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T09:11:18.299 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:18 ntpd[28020]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T09:11:18.299 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:18 ntpd[28020]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T09:11:18.300 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:18 ntpd[28020]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T09:11:18.300 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:18 ntpd[28020]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T09:11:18.300 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:18 ntpd[28020]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T09:11:19.299 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:19 ntpd[28020]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T09:11:19.299 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:19 ntpd[28020]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T09:11:19.300 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:19 ntpd[28020]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T09:11:19.300 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:19 ntpd[28020]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T09:11:19.300 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:19 ntpd[28020]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T09:11:19.300 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:19 ntpd[28020]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T09:11:19.300 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:19 ntpd[28020]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T09:11:19.301 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:19 ntpd[28020]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T09:11:19.301 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:19 ntpd[28020]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T09:11:19.301 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:19 ntpd[28020]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T09:11:19.301 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:19 ntpd[28020]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T09:11:19.301 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:19 ntpd[28020]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T09:11:20.299 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:20 ntpd[28020]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T09:11:20.299 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:20 ntpd[28020]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T09:11:20.299 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:20 ntpd[28020]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T09:11:20.300 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:20 ntpd[28020]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T09:11:20.300 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:20 ntpd[28020]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T09:11:20.300 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:20 ntpd[28020]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T09:11:20.300 INFO:teuthology.orchestra.run.smithi150.stdout:20 Oct 09:11:20 ntpd[28020]: ntpd: time slew +0.000652 s 2023-10-20T09:11:20.301 INFO:teuthology.orchestra.run.smithi150.stdout:ntpd: time slew +0.000652s 2023-10-20T09:11:20.301 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:20 ntpd[28020]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-20T09:11:20.301 INFO:teuthology.orchestra.run.smithi150.stderr:20 Oct 09:11:20 ntpd[28020]: can't open /var/log/ntpstats/loopstats.20231020: Permission denied 2023-10-20T09:11:20.376 INFO:teuthology.orchestra.run.smithi150.stdout: remote refid st t when poll reach delay offset jitter 2023-10-20T09:11:20.376 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================== 2023-10-20T09:11:20.377 INFO:teuthology.orchestra.run.smithi150.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-20T09:11:20.377 INFO:teuthology.orchestra.run.smithi150.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-20T09:11:20.377 INFO:teuthology.orchestra.run.smithi150.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-20T09:11:20.378 INFO:teuthology.orchestra.run.smithi150.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-20T09:11:20.379 INFO:teuthology.run_tasks:Running task install... 2023-10-20T09:11:20.391 DEBUG:teuthology.task.install:project ceph 2023-10-20T09:11:20.392 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'}} 2023-10-20T09:11:20.393 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'} 2023-10-20T09:11:20.393 INFO:teuthology.task.install:Using flavor: default 2023-10-20T09:11:20.401 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-20T09:11:20.402 INFO:teuthology.task.install:extra packages: [] 2023-10-20T09:11:20.403 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-key list | grep Ceph 2023-10-20T09:11:20.480 INFO:teuthology.orchestra.run.smithi150.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-20T09:11:20.508 INFO:teuthology.orchestra.run.smithi150.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-20T09:11:20.508 INFO:teuthology.orchestra.run.smithi150.stdout:uid [ unknown] Ceph.com (release key) 2023-10-20T09:11:20.508 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-20T09:11:20.509 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-20T09:11:20.732 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default/ 2023-10-20T09:11:20.733 INFO:teuthology.task.install.deb:Package version is 18.0.0-6744-g7f103c7c-1jammy 2023-10-20T09:11:21.049 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-20T09:11:21.049 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-20T09:11:21.064 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-get update 2023-10-20T09:11:21.235 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-20T09:11:21.353 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-20T09:11:21.437 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-20T09:11:21.502 INFO:teuthology.orchestra.run.smithi150.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy InRelease 2023-10-20T09:11:21.516 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-20T09:11:21.582 INFO:teuthology.orchestra.run.smithi150.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy Release [6,978 B] 2023-10-20T09:11:21.662 INFO:teuthology.orchestra.run.smithi150.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-20T09:11:21.740 INFO:teuthology.orchestra.run.smithi150.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,417 B] 2023-10-20T09:11:21.829 INFO:teuthology.orchestra.run.smithi150.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-10-20T09:11:21.913 INFO:teuthology.orchestra.run.smithi150.stdout:Fetched 27.9 kB in 1s (41.9 kB/s) 2023-10-20T09:11:22.778 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2023-10-20T09:11:22.794 DEBUG:teuthology.orchestra.run.smithi150:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6744-g7f103c7c-1jammy cephadm=18.0.0-6744-g7f103c7c-1jammy ceph-mds=18.0.0-6744-g7f103c7c-1jammy ceph-mgr=18.0.0-6744-g7f103c7c-1jammy ceph-common=18.0.0-6744-g7f103c7c-1jammy ceph-fuse=18.0.0-6744-g7f103c7c-1jammy ceph-test=18.0.0-6744-g7f103c7c-1jammy ceph-volume=18.0.0-6744-g7f103c7c-1jammy radosgw=18.0.0-6744-g7f103c7c-1jammy python3-rados=18.0.0-6744-g7f103c7c-1jammy python3-rgw=18.0.0-6744-g7f103c7c-1jammy python3-cephfs=18.0.0-6744-g7f103c7c-1jammy python3-rbd=18.0.0-6744-g7f103c7c-1jammy libcephfs2=18.0.0-6744-g7f103c7c-1jammy libcephfs-dev=18.0.0-6744-g7f103c7c-1jammy librados2=18.0.0-6744-g7f103c7c-1jammy librbd1=18.0.0-6744-g7f103c7c-1jammy rbd-fuse=18.0.0-6744-g7f103c7c-1jammy 2023-10-20T09:11:22.854 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2023-10-20T09:11:23.042 INFO:teuthology.orchestra.run.smithi150.stdout:Building dependency tree... 2023-10-20T09:11:23.042 INFO:teuthology.orchestra.run.smithi150.stdout:Reading state information... 2023-10-20T09:11:23.202 INFO:teuthology.orchestra.run.smithi150.stdout:The following packages were automatically installed and are no longer required: 2023-10-20T09:11:23.203 INFO:teuthology.orchestra.run.smithi150.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-20T09:11:23.203 INFO:teuthology.orchestra.run.smithi150.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-20T09:11:23.203 INFO:teuthology.orchestra.run.smithi150.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-20T09:11:23.203 INFO:teuthology.orchestra.run.smithi150.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-20T09:11:23.203 INFO:teuthology.orchestra.run.smithi150.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-20T09:11:23.204 INFO:teuthology.orchestra.run.smithi150.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-20T09:11:23.205 INFO:teuthology.orchestra.run.smithi150.stdout:The following additional packages will be installed: 2023-10-20T09:11:23.205 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-20T09:11:23.205 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-20T09:11:23.205 INFO:teuthology.orchestra.run.smithi150.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-20T09:11:23.206 INFO:teuthology.orchestra.run.smithi150.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-20T09:11:23.206 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-20T09:11:23.206 INFO:teuthology.orchestra.run.smithi150.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-20T09:11:23.206 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-20T09:11:23.206 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-20T09:11:23.207 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-10-20T09:11:23.207 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-20T09:11:23.207 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-20T09:11:23.207 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-20T09:11:23.207 INFO:teuthology.orchestra.run.smithi150.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-20T09:11:23.208 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-20T09:11:23.208 INFO:teuthology.orchestra.run.smithi150.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-20T09:11:23.208 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-10-20T09:11:23.208 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-10-20T09:11:23.208 INFO:teuthology.orchestra.run.smithi150.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-20T09:11:23.209 INFO:teuthology.orchestra.run.smithi150.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-20T09:11:23.209 INFO:teuthology.orchestra.run.smithi150.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-20T09:11:23.209 INFO:teuthology.orchestra.run.smithi150.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-20T09:11:23.209 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-10-20T09:11:23.209 INFO:teuthology.orchestra.run.smithi150.stdout:Suggested packages: 2023-10-20T09:11:23.210 INFO:teuthology.orchestra.run.smithi150.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-20T09:11:23.210 INFO:teuthology.orchestra.run.smithi150.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-20T09:11:23.210 INFO:teuthology.orchestra.run.smithi150.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-20T09:11:23.210 INFO:teuthology.orchestra.run.smithi150.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-20T09:11:23.210 INFO:teuthology.orchestra.run.smithi150.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-20T09:11:23.211 INFO:teuthology.orchestra.run.smithi150.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-20T09:11:23.211 INFO:teuthology.orchestra.run.smithi150.stdout: python3-watchdog 2023-10-20T09:11:23.211 INFO:teuthology.orchestra.run.smithi150.stdout:Recommended packages: 2023-10-20T09:11:23.211 INFO:teuthology.orchestra.run.smithi150.stdout: btrfs-tools 2023-10-20T09:11:23.299 INFO:teuthology.orchestra.run.smithi150.stdout:The following NEW packages will be installed: 2023-10-20T09:11:23.300 INFO:teuthology.orchestra.run.smithi150.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-20T09:11:23.300 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-20T09:11:23.300 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-10-20T09:11:23.301 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-10-20T09:11:23.301 INFO:teuthology.orchestra.run.smithi150.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-10-20T09:11:23.301 INFO:teuthology.orchestra.run.smithi150.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-20T09:11:23.302 INFO:teuthology.orchestra.run.smithi150.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-20T09:11:23.302 INFO:teuthology.orchestra.run.smithi150.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-20T09:11:23.302 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-20T09:11:23.303 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-10-20T09:11:23.303 INFO:teuthology.orchestra.run.smithi150.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-20T09:11:23.303 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-20T09:11:23.304 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-20T09:11:23.304 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-20T09:11:23.304 INFO:teuthology.orchestra.run.smithi150.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-20T09:11:23.304 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-20T09:11:23.305 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-20T09:11:23.305 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-20T09:11:23.305 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-20T09:11:23.306 INFO:teuthology.orchestra.run.smithi150.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-20T09:11:23.306 INFO:teuthology.orchestra.run.smithi150.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-20T09:11:23.307 INFO:teuthology.orchestra.run.smithi150.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-20T09:11:23.307 INFO:teuthology.orchestra.run.smithi150.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-20T09:11:23.307 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-20T09:11:23.307 INFO:teuthology.orchestra.run.smithi150.stdout: socat xmlstarlet 2023-10-20T09:11:23.376 INFO:teuthology.orchestra.run.smithi150.stdout:0 upgraded, 109 newly installed, 0 to remove and 257 not upgraded. 2023-10-20T09:11:23.377 INFO:teuthology.orchestra.run.smithi150.stdout:Need to get 184 MB of archives. 2023-10-20T09:11:23.377 INFO:teuthology.orchestra.run.smithi150.stdout:After this operation, 824 MB of additional disk space will be used. 2023-10-20T09:11:23.377 INFO:teuthology.orchestra.run.smithi150.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-20T09:11:23.444 INFO:teuthology.orchestra.run.smithi150.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-20T09:11:23.485 INFO:teuthology.orchestra.run.smithi150.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-20T09:11:23.489 INFO:teuthology.orchestra.run.smithi150.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-20T09:11:23.507 INFO:teuthology.orchestra.run.smithi150.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-20T09:11:23.527 INFO:teuthology.orchestra.run.smithi150.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6744-g7f103c7c-1jammy [3,450 kB] 2023-10-20T09:11:23.564 INFO:teuthology.orchestra.run.smithi150.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-20T09:11:23.568 INFO:teuthology.orchestra.run.smithi150.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-20T09:11:23.580 INFO:teuthology.orchestra.run.smithi150.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-20T09:11:23.584 INFO:teuthology.orchestra.run.smithi150.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-20T09:11:23.585 INFO:teuthology.orchestra.run.smithi150.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-20T09:11:23.586 INFO:teuthology.orchestra.run.smithi150.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-20T09:11:23.594 INFO:teuthology.orchestra.run.smithi150.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-20T09:11:23.595 INFO:teuthology.orchestra.run.smithi150.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-20T09:11:23.595 INFO:teuthology.orchestra.run.smithi150.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-20T09:11:23.596 INFO:teuthology.orchestra.run.smithi150.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-20T09:11:23.608 INFO:teuthology.orchestra.run.smithi150.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-20T09:11:23.609 INFO:teuthology.orchestra.run.smithi150.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-20T09:11:23.609 INFO:teuthology.orchestra.run.smithi150.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-20T09:11:23.610 INFO:teuthology.orchestra.run.smithi150.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-20T09:11:23.628 INFO:teuthology.orchestra.run.smithi150.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-20T09:11:23.628 INFO:teuthology.orchestra.run.smithi150.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-20T09:11:23.631 INFO:teuthology.orchestra.run.smithi150.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-20T09:11:23.631 INFO:teuthology.orchestra.run.smithi150.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-20T09:11:23.632 INFO:teuthology.orchestra.run.smithi150.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-20T09:11:23.632 INFO:teuthology.orchestra.run.smithi150.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-20T09:11:23.648 INFO:teuthology.orchestra.run.smithi150.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-20T09:11:23.648 INFO:teuthology.orchestra.run.smithi150.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-20T09:11:23.648 INFO:teuthology.orchestra.run.smithi150.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-20T09:11:23.650 INFO:teuthology.orchestra.run.smithi150.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-20T09:11:23.670 INFO:teuthology.orchestra.run.smithi150.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-20T09:11:23.670 INFO:teuthology.orchestra.run.smithi150.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-20T09:11:23.674 INFO:teuthology.orchestra.run.smithi150.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-20T09:11:23.674 INFO:teuthology.orchestra.run.smithi150.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-20T09:11:23.674 INFO:teuthology.orchestra.run.smithi150.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-20T09:11:23.675 INFO:teuthology.orchestra.run.smithi150.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-20T09:11:23.687 INFO:teuthology.orchestra.run.smithi150.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-20T09:11:23.689 INFO:teuthology.orchestra.run.smithi150.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-20T09:11:23.690 INFO:teuthology.orchestra.run.smithi150.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-20T09:11:23.691 INFO:teuthology.orchestra.run.smithi150.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-20T09:11:23.706 INFO:teuthology.orchestra.run.smithi150.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-20T09:11:23.709 INFO:teuthology.orchestra.run.smithi150.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-20T09:11:23.709 INFO:teuthology.orchestra.run.smithi150.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-20T09:11:23.712 INFO:teuthology.orchestra.run.smithi150.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-20T09:11:23.784 INFO:teuthology.orchestra.run.smithi150.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-20T09:11:23.785 INFO:teuthology.orchestra.run.smithi150.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-20T09:11:23.787 INFO:teuthology.orchestra.run.smithi150.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-20T09:11:23.788 INFO:teuthology.orchestra.run.smithi150.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-20T09:11:23.788 INFO:teuthology.orchestra.run.smithi150.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-20T09:11:23.789 INFO:teuthology.orchestra.run.smithi150.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-20T09:11:23.828 INFO:teuthology.orchestra.run.smithi150.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-20T09:11:23.831 INFO:teuthology.orchestra.run.smithi150.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-20T09:11:23.831 INFO:teuthology.orchestra.run.smithi150.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-20T09:11:23.859 INFO:teuthology.orchestra.run.smithi150.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-20T09:11:23.860 INFO:teuthology.orchestra.run.smithi150.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-20T09:11:23.861 INFO:teuthology.orchestra.run.smithi150.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-20T09:11:23.862 INFO:teuthology.orchestra.run.smithi150.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-20T09:11:23.863 INFO:teuthology.orchestra.run.smithi150.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-20T09:11:23.864 INFO:teuthology.orchestra.run.smithi150.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-20T09:11:23.867 INFO:teuthology.orchestra.run.smithi150.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-20T09:11:23.868 INFO:teuthology.orchestra.run.smithi150.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-20T09:11:23.872 INFO:teuthology.orchestra.run.smithi150.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-20T09:11:23.886 INFO:teuthology.orchestra.run.smithi150.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-20T09:11:23.888 INFO:teuthology.orchestra.run.smithi150.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-20T09:11:23.889 INFO:teuthology.orchestra.run.smithi150.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-20T09:11:23.893 INFO:teuthology.orchestra.run.smithi150.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-20T09:11:23.896 INFO:teuthology.orchestra.run.smithi150.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-20T09:11:23.901 INFO:teuthology.orchestra.run.smithi150.stdout:Get:68 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6744-g7f103c7c-1jammy [3,132 kB] 2023-10-20T09:11:23.902 INFO:teuthology.orchestra.run.smithi150.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-20T09:11:23.908 INFO:teuthology.orchestra.run.smithi150.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-20T09:11:23.908 INFO:teuthology.orchestra.run.smithi150.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-20T09:11:23.909 INFO:teuthology.orchestra.run.smithi150.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-20T09:11:23.909 INFO:teuthology.orchestra.run.smithi150.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-20T09:11:23.925 INFO:teuthology.orchestra.run.smithi150.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-20T09:11:23.926 INFO:teuthology.orchestra.run.smithi150.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-20T09:11:23.936 INFO:teuthology.orchestra.run.smithi150.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-20T09:11:23.936 INFO:teuthology.orchestra.run.smithi150.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-20T09:11:23.949 INFO:teuthology.orchestra.run.smithi150.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-20T09:11:23.951 INFO:teuthology.orchestra.run.smithi150.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-20T09:11:23.952 INFO:teuthology.orchestra.run.smithi150.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-20T09:11:24.004 INFO:teuthology.orchestra.run.smithi150.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6744-g7f103c7c-1jammy [689 kB] 2023-10-20T09:11:24.022 INFO:teuthology.orchestra.run.smithi150.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6744-g7f103c7c-1jammy [356 kB] 2023-10-20T09:11:24.031 INFO:teuthology.orchestra.run.smithi150.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6744-g7f103c7c-1jammy [32.8 kB] 2023-10-20T09:11:24.032 INFO:teuthology.orchestra.run.smithi150.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6744-g7f103c7c-1jammy [180 kB] 2023-10-20T09:11:24.035 INFO:teuthology.orchestra.run.smithi150.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6744-g7f103c7c-1jammy [62.4 kB] 2023-10-20T09:11:24.038 INFO:teuthology.orchestra.run.smithi150.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6744-g7f103c7c-1jammy [330 kB] 2023-10-20T09:11:24.049 INFO:teuthology.orchestra.run.smithi150.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6744-g7f103c7c-1jammy [6,221 kB] 2023-10-20T09:11:24.255 INFO:teuthology.orchestra.run.smithi150.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6744-g7f103c7c-1jammy [112 kB] 2023-10-20T09:11:24.257 INFO:teuthology.orchestra.run.smithi150.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6744-g7f103c7c-1jammy [441 kB] 2023-10-20T09:11:24.271 INFO:teuthology.orchestra.run.smithi150.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6744-g7f103c7c-1jammy [22.6 MB] 2023-10-20T09:11:25.165 INFO:teuthology.orchestra.run.smithi150.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6744-g7f103c7c-1jammy [6,852 kB] 2023-10-20T09:11:25.298 INFO:teuthology.orchestra.run.smithi150.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6744-g7f103c7c-1jammy [238 kB] 2023-10-20T09:11:25.304 INFO:teuthology.orchestra.run.smithi150.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6744-g7f103c7c-1jammy [128 kB] 2023-10-20T09:11:25.310 INFO:teuthology.orchestra.run.smithi150.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6744-g7f103c7c-1jammy [1,496 kB] 2023-10-20T09:11:25.358 INFO:teuthology.orchestra.run.smithi150.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6744-g7f103c7c-1jammy [9,863 kB] 2023-10-20T09:11:25.683 INFO:teuthology.orchestra.run.smithi150.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6744-g7f103c7c-1jammy [31.7 MB] 2023-10-20T09:11:27.177 INFO:teuthology.orchestra.run.smithi150.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6744-g7f103c7c-1jammy [14.0 kB] 2023-10-20T09:11:27.178 INFO:teuthology.orchestra.run.smithi150.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6744-g7f103c7c-1jammy [839 kB] 2023-10-20T09:11:27.194 INFO:teuthology.orchestra.run.smithi150.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6744-g7f103c7c-1jammy [2,146 kB] 2023-10-20T09:11:27.232 INFO:teuthology.orchestra.run.smithi150.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6744-g7f103c7c-1jammy [283 kB] 2023-10-20T09:11:27.238 INFO:teuthology.orchestra.run.smithi150.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6744-g7f103c7c-1jammy [138 kB] 2023-10-20T09:11:27.243 INFO:teuthology.orchestra.run.smithi150.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6744-g7f103c7c-1jammy [1,906 kB] 2023-10-20T09:11:27.351 INFO:teuthology.orchestra.run.smithi150.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6744-g7f103c7c-1jammy [8,625 kB] 2023-10-20T09:11:28.089 INFO:teuthology.orchestra.run.smithi150.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6744-g7f103c7c-1jammy [14.2 kB] 2023-10-20T09:11:28.136 INFO:teuthology.orchestra.run.smithi150.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6744-g7f103c7c-1jammy [47.8 MB] 2023-10-20T09:11:33.292 INFO:teuthology.orchestra.run.smithi150.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6744-g7f103c7c-1jammy [120 kB] 2023-10-20T09:11:33.294 INFO:teuthology.orchestra.run.smithi150.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6744-g7f103c7c-1jammy [38.3 kB] 2023-10-20T09:11:33.295 INFO:teuthology.orchestra.run.smithi150.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6744-g7f103c7c-1jammy [12.3 MB] 2023-10-20T09:11:35.261 INFO:teuthology.orchestra.run.smithi150.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6744-g7f103c7c-1jammy [91.6 kB] 2023-10-20T09:11:35.809 INFO:teuthology.orchestra.run.smithi150.stdout:Fetched 184 MB in 12s (15.4 MB/s) 2023-10-20T09:11:36.058 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-20T09:11:36.160 INFO:teuthology.orchestra.run.smithi150.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165984 files and directories currently installed.) 2023-10-20T09:11:36.165 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-20T09:11:36.232 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-20T09:11:36.568 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-20T09:11:36.594 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-20T09:11:36.635 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-20T09:11:36.980 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-20T09:11:37.006 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-20T09:11:37.047 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-20T09:11:37.383 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-20T09:11:37.409 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-20T09:11:37.449 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-20T09:11:37.895 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-20T09:11:37.921 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-20T09:11:37.970 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T09:11:38.356 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-20T09:11:38.382 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-20T09:11:38.422 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T09:11:38.742 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-20T09:11:38.769 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-20T09:11:38.801 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T09:11:39.171 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-20T09:11:39.197 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-20T09:11:39.237 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-20T09:11:39.565 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package librados2. 2023-10-20T09:11:39.591 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../008-librados2_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-20T09:11:39.631 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking librados2 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:11:40.068 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package librbd1. 2023-10-20T09:11:40.094 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../009-librbd1_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-20T09:11:40.135 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking librbd1 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:11:40.554 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libcephfs2. 2023-10-20T09:11:40.580 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-20T09:11:40.621 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libcephfs2 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:11:40.932 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-rados. 2023-10-20T09:11:40.958 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-20T09:11:41.173 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-rados (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:11:41.509 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-20T09:11:41.536 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-20T09:11:41.576 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-ceph-argparse (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:11:41.770 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-cephfs. 2023-10-20T09:11:41.785 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-20T09:11:41.828 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-cephfs (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:11:42.148 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-20T09:11:42.174 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-20T09:11:42.214 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-ceph-common (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:11:42.543 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-20T09:11:42.569 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-20T09:11:42.609 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-20T09:11:42.929 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-prettytable. 2023-10-20T09:11:42.955 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-10-20T09:11:42.995 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-20T09:11:43.248 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-rbd. 2023-10-20T09:11:43.274 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-20T09:11:43.306 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-rbd (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:11:43.651 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-20T09:11:43.677 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-20T09:11:43.717 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-20T09:11:44.054 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package librgw2. 2023-10-20T09:11:44.080 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../019-librgw2_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-20T09:11:44.120 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking librgw2 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:11:44.648 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-rgw. 2023-10-20T09:11:44.663 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-20T09:11:44.698 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-rgw (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:11:45.001 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-20T09:11:45.027 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-20T09:11:45.067 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-20T09:11:45.412 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libradosstriper1. 2023-10-20T09:11:45.438 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-20T09:11:45.478 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libradosstriper1 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:11:45.923 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-common. 2023-10-20T09:11:45.950 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-20T09:11:45.990 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-common (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:11:47.288 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-base. 2023-10-20T09:11:47.314 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-20T09:11:47.363 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-base (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:11:48.008 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-20T09:11:48.035 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-20T09:11:48.075 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-20T09:11:48.378 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-cheroot. 2023-10-20T09:11:48.404 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-20T09:11:48.444 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-20T09:11:48.764 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-20T09:11:48.790 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-20T09:11:48.831 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-20T09:11:49.133 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-20T09:11:49.160 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-20T09:11:49.200 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-20T09:11:49.503 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-20T09:11:49.530 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-20T09:11:49.561 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-20T09:11:49.872 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-tempora. 2023-10-20T09:11:49.899 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-10-20T09:11:49.939 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-20T09:11:50.242 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-portend. 2023-10-20T09:11:50.269 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-10-20T09:11:50.317 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-20T09:11:50.595 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-20T09:11:50.621 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-20T09:11:50.661 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-20T09:11:50.931 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-20T09:11:50.957 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-20T09:11:50.998 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-20T09:11:51.526 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-natsort. 2023-10-20T09:11:51.542 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-10-20T09:11:51.576 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-20T09:11:51.854 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-logutils. 2023-10-20T09:11:51.881 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-10-20T09:11:51.921 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-20T09:11:52.224 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-20T09:11:52.250 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-20T09:11:52.290 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-20T09:11:52.593 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-mako. 2023-10-20T09:11:52.620 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-20T09:11:52.660 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-20T09:11:52.988 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-20T09:11:53.014 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-20T09:11:53.054 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-20T09:11:53.382 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-20T09:11:53.409 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-20T09:11:53.449 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-20T09:11:53.737 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-webob. 2023-10-20T09:11:53.752 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-20T09:11:53.787 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-20T09:11:54.115 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-waitress. 2023-10-20T09:11:54.141 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-20T09:11:54.185 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-20T09:11:54.493 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-tempita. 2023-10-20T09:11:54.519 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-20T09:11:54.559 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-20T09:11:54.904 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-paste. 2023-10-20T09:11:54.931 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-20T09:11:54.979 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-20T09:11:55.298 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-20T09:11:55.314 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-20T09:11:55.357 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-20T09:11:55.652 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-20T09:11:55.679 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-20T09:11:55.718 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-20T09:11:56.038 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-webtest. 2023-10-20T09:11:56.065 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-10-20T09:11:56.104 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-20T09:11:57.175 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pecan. 2023-10-20T09:11:57.202 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-20T09:11:57.233 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-20T09:11:57.571 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-certifi. 2023-10-20T09:11:57.598 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-10-20T09:11:57.646 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-20T09:11:57.973 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-idna. 2023-10-20T09:11:58.000 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-10-20T09:11:58.040 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-20T09:11:58.351 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-urllib3. 2023-10-20T09:11:58.378 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-20T09:11:58.418 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-20T09:11:58.746 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-requests. 2023-10-20T09:11:58.773 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-20T09:11:58.813 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-20T09:11:59.107 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-20T09:11:59.134 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-20T09:11:59.174 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-20T09:11:59.460 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-20T09:11:59.475 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-20T09:11:59.518 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:11:59.898 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-20T09:11:59.925 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-20T09:11:59.964 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:00.284 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-openssl. 2023-10-20T09:12:00.311 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-10-20T09:12:00.350 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-20T09:12:00.695 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mgr. 2023-10-20T09:12:00.722 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-20T09:12:00.762 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mgr (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:01.123 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mon. 2023-10-20T09:12:01.139 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-20T09:12:01.173 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mon (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:02.293 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-osd. 2023-10-20T09:12:02.320 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-20T09:12:02.360 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-osd (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:04.229 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph. 2023-10-20T09:12:04.245 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../059-ceph_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-20T09:12:04.288 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:04.582 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-fuse. 2023-10-20T09:12:04.609 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-20T09:12:04.641 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-fuse (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:04.994 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mds. 2023-10-20T09:12:05.020 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-20T09:12:05.060 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mds (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:05.455 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package cephadm. 2023-10-20T09:12:05.482 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../062-cephadm_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-20T09:12:05.521 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking cephadm (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:05.841 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-20T09:12:05.868 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-20T09:12:05.907 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-20T09:12:06.210 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-20T09:12:06.226 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-20T09:12:06.269 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-20T09:12:07.308 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-babel. 2023-10-20T09:12:07.335 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-20T09:12:07.374 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-20T09:12:07.719 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-jinja2. 2023-10-20T09:12:07.746 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-10-20T09:12:07.785 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-20T09:12:08.139 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-20T09:12:08.166 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-20T09:12:08.205 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:08.550 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-20T09:12:08.577 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-20T09:12:08.617 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-20T09:12:08.920 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-20T09:12:08.946 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-10-20T09:12:08.986 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-20T09:12:09.364 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-routes. 2023-10-20T09:12:09.392 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-20T09:12:09.439 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-20T09:12:09.776 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-20T09:12:09.803 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-20T09:12:09.842 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:10.304 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-20T09:12:10.320 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-20T09:12:10.362 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-20T09:12:10.849 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-joblib. 2023-10-20T09:12:10.865 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-20T09:12:10.907 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-20T09:12:11.253 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-20T09:12:11.280 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-20T09:12:11.319 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-20T09:12:11.622 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-sklearn. 2023-10-20T09:12:11.650 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-20T09:12:11.689 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-20T09:12:12.648 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-20T09:12:12.675 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-20T09:12:12.714 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:13.767 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-cachetools. 2023-10-20T09:12:13.794 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-10-20T09:12:13.842 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-20T09:12:14.236 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-20T09:12:14.263 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-20T09:12:14.311 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-20T09:12:14.690 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-20T09:12:14.716 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-20T09:12:14.756 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-20T09:12:15.076 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-rsa. 2023-10-20T09:12:15.103 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-10-20T09:12:15.142 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-20T09:12:15.454 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-google-auth. 2023-10-20T09:12:15.481 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-10-20T09:12:15.529 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-20T09:12:15.840 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-20T09:12:15.867 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-20T09:12:15.898 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-20T09:12:16.253 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-websocket. 2023-10-20T09:12:16.280 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-10-20T09:12:16.320 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-20T09:12:16.640 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-20T09:12:16.667 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-20T09:12:16.706 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-20T09:12:17.479 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-20T09:12:17.506 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-20T09:12:17.545 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:17.898 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-20T09:12:17.925 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-20T09:12:17.964 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-20T09:12:18.301 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-20T09:12:18.328 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-20T09:12:18.359 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-20T09:12:18.603 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package jq. 2023-10-20T09:12:18.619 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-20T09:12:18.661 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-20T09:12:18.931 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package socat. 2023-10-20T09:12:18.947 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-20T09:12:18.989 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-20T09:12:19.284 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package xmlstarlet. 2023-10-20T09:12:19.300 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-20T09:12:19.350 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-20T09:12:19.713 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-test. 2023-10-20T09:12:19.738 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-20T09:12:19.780 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-test (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:22.034 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-volume. 2023-10-20T09:12:22.062 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6744-g7f103c7c-1jammy_all.deb ... 2023-10-20T09:12:22.100 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-volume (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:22.446 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-20T09:12:22.462 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-20T09:12:22.504 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libcephfs-dev (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:23.024 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package nvme-cli. 2023-10-20T09:12:23.051 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-20T09:12:23.090 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-20T09:12:23.420 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-20T09:12:23.436 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-20T09:12:23.470 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-20T09:12:23.924 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-20T09:12:23.951 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-20T09:12:23.982 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-20T09:12:24.268 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pastescript. 2023-10-20T09:12:24.296 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-10-20T09:12:24.334 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-20T09:12:24.655 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pluggy. 2023-10-20T09:12:24.683 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-20T09:12:24.722 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-20T09:12:25.008 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-psutil. 2023-10-20T09:12:25.035 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-20T09:12:25.074 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-20T09:12:25.427 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-py. 2023-10-20T09:12:25.454 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-10-20T09:12:25.502 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-20T09:12:25.830 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pygments. 2023-10-20T09:12:25.858 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-20T09:12:25.897 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-20T09:12:26.376 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-20T09:12:26.404 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-20T09:12:26.434 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-20T09:12:26.745 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-toml. 2023-10-20T09:12:26.773 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-10-20T09:12:26.811 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-20T09:12:27.123 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pytest. 2023-10-20T09:12:27.151 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-20T09:12:27.189 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-20T09:12:27.534 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-simplejson. 2023-10-20T09:12:27.562 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-20T09:12:27.601 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-20T09:12:27.870 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-20T09:12:27.886 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-20T09:12:27.928 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-20T09:12:28.724 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package radosgw. 2023-10-20T09:12:28.752 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../107-radosgw_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-20T09:12:28.790 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking radosgw (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:29.543 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package rbd-fuse. 2023-10-20T09:12:29.571 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6744-g7f103c7c-1jammy_amd64.deb ... 2023-10-20T09:12:29.610 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking rbd-fuse (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:29.913 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-20T09:12:30.165 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-20T09:12:30.266 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-20T09:12:30.531 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-20T09:12:30.860 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-20T09:12:31.332 INFO:teuthology.orchestra.run.smithi150.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-20T09:12:31.434 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up cephadm (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:31.977 INFO:teuthology.orchestra.run.smithi150.stdout:Adding system user cephadm....done 2023-10-20T09:12:32.115 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-20T09:12:32.367 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-20T09:12:32.601 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-20T09:12:32.727 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-20T09:12:32.978 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-20T09:12:33.454 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-20T09:12:33.572 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-20T09:12:33.874 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-20T09:12:34.201 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-20T09:12:34.452 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-20T09:12:34.745 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-20T09:12:34.995 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-20T09:12:35.272 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-20T09:12:35.515 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-ceph-argparse (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:35.859 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-20T09:12:36.058 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-20T09:12:36.310 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-20T09:12:36.436 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-20T09:12:36.795 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-20T09:12:36.921 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-20T09:12:37.191 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-20T09:12:37.433 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-20T09:12:37.735 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-20T09:12:38.437 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-20T09:12:38.696 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-20T09:12:38.822 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-20T09:12:38.948 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-20T09:12:39.241 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-20T09:12:39.642 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-20T09:12:39.885 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-20T09:12:40.253 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-20T09:12:40.546 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-20T09:12:40.880 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-20T09:12:41.123 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-20T09:12:41.233 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-ceph-common (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:41.518 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-idna (3.3-1) ... 2023-10-20T09:12:41.835 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-20T09:12:43.041 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T09:12:43.176 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-20T09:12:43.597 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-20T09:12:43.728 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-20T09:12:43.854 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-20T09:12:44.163 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-20T09:12:44.398 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-20T09:12:44.641 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T09:12:44.767 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-20T09:12:44.892 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-20T09:12:45.194 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-20T09:12:45.453 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-20T09:12:45.713 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-20T09:12:45.947 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-20T09:12:46.182 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-20T09:12:46.424 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-20T09:12:46.550 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-20T09:12:46.835 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-20T09:12:46.961 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-20T09:12:47.254 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-20T09:12:47.546 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-20T09:12:47.781 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-20T09:12:48.169 INFO:teuthology.orchestra.run.smithi150.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-20T09:12:48.417 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-20T09:12:48.543 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-20T09:12:48.945 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-20T09:12:49.213 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-20T09:12:49.439 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T09:12:49.540 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-20T09:12:49.841 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-20T09:12:50.150 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-20T09:12:50.443 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-20T09:12:50.569 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-20T09:12:50.853 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-20T09:12:51.130 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-20T09:12:51.364 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-20T09:12:51.757 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-20T09:12:52.025 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-20T09:12:52.369 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-20T09:12:52.495 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up librados2 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:52.621 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up librgw2 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:52.747 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:52.873 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libcephfs2 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:52.991 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libradosstriper1 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:53.210 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-20T09:12:53.496 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up librbd1 (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:53.621 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mgr-modules-core (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:53.748 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-fuse (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:53.959 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-20T09:12:53.959 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-20T09:12:54.499 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libcephfs-dev (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:54.600 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-rados (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:54.709 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-20T09:12:55.864 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-rbd (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:55.990 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up rbd-fuse (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:56.108 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-rgw (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:56.209 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-cephfs (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:56.318 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-common (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:56.869 INFO:teuthology.orchestra.run.smithi150.stdout:Adding group ceph....done 2023-10-20T09:12:57.279 INFO:teuthology.orchestra.run.smithi150.stdout:Adding system user ceph....done 2023-10-20T09:12:57.423 INFO:teuthology.orchestra.run.smithi150.stdout:Setting system user ceph properties....done 2023-10-20T09:12:57.442 INFO:teuthology.orchestra.run.smithi150.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-20T09:12:57.582 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-20T09:12:57.955 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-20T09:12:58.478 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-test (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:58.604 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up radosgw (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:59.071 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-20T09:12:59.071 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-20T09:12:59.572 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-base (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:12:59.898 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-20T09:13:00.432 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mds (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:13:00.653 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-20T09:13:00.653 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-20T09:13:01.126 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mgr (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:13:01.352 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-20T09:13:01.352 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-20T09:13:01.894 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-osd (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:13:02.197 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-20T09:13:02.198 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-20T09:13:02.728 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:13:02.854 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:13:02.997 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mon (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:13:03.193 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-20T09:13:03.194 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-20T09:13:03.732 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mgr-cephadm (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:13:03.892 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:13:04.276 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mgr-dashboard (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:13:04.427 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-volume (18.0.0-6744-g7f103c7c-1jammy) ... 2023-10-20T09:13:04.788 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-20T09:13:05.972 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-20T09:13:06.459 INFO:teuthology.orchestra.run.smithi150.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-20T09:13:06.462 DEBUG:teuthology.parallel:result is None 2023-10-20T09:13:06.462 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-20T09:13:06.688 DEBUG:teuthology.orchestra.run.smithi150:> dpkg-query -W -f '${Version}' ceph 2023-10-20T09:13:06.708 INFO:teuthology.orchestra.run.smithi150.stdout:18.0.0-6744-g7f103c7c-1jammy 2023-10-20T09:13:06.709 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6744-g7f103c7c-1jammy 2023-10-20T09:13:06.709 INFO:teuthology.task.install:The correct ceph version 18.0.0-6744-g7f103c7c-1jammy is installed. 2023-10-20T09:13:06.709 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-20T09:13:06.710 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-20T09:13:06.711 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-20T09:13:06.765 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-20T09:13:06.766 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-20T09:13:06.766 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/daemon-helper 2023-10-20T09:13:06.819 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-20T09:13:06.876 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-20T09:13:06.876 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-20T09:13:06.876 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-20T09:13:06.932 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-20T09:13:06.992 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-20T09:13:06.992 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-20T09:13:06.993 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/stdin-killer 2023-10-20T09:13:07.047 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-20T09:13:07.103 INFO:teuthology.run_tasks:Running task ceph... 2023-10-20T09:13:07.183 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-20T09:13:07.184 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 777 /var/log/ceph 2023-10-20T09:13:07.197 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-20T09:13:07.197 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-20T09:13:07.252 INFO:tasks.ceph:Creating extra log directories... 2023-10-20T09:13:07.252 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-20T09:13:07.314 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-20T09:13:07.315 INFO:tasks.ceph:config {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 1}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-20T09:13:07.315 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rraja-2023-10-20_08:52:26-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7432983', 'branch': 'ajarr-wip-extra-debug-62994-oct-19', 'description': 'rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery}', 'email': 'rraja@redhat.com', 'first_in_suite': False, 'job_id': '7432983', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'rraja-2023-10-20_08:52:26-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'ajarr-wip-extra-debug-62994-oct-19'}, 'ceph': {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 1}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {'osd default pool size': 2}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'ajarr-wip-extra-debug-62994-oct-19', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'}}, 'owner': 'scheduled_rraja@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 9138, 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab', 'sleep_before_teardown': 0, 'suite': 'rbd:cli', 'suite_branch': 'ajarr-wip-extra-debug-62994-oct-19', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab', 'targets': {'smithi150.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMhNZKSaorwOILbIWwC8yum4ac4p9BDJPPbLNv44G7t2YNONzis8NtcZXgBhQBBlI9Fi9CxItahxOgAuWR2Gut0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'exec': {'client.0': ['sudo ceph osd pool create datapool 4', 'rbd pool init datapool']}}, {'install': {'extra_system_packages': ['fio']}}, {'workunit': {'clients': {'client.0': ['rbd/rbd_support_module_recovery.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '202b180cb047e798fb131047314a862593f45403', 'timestamp': '2023-10-20_08:52:26', 'tube': 'smithi', 'user': 'rraja', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-20T09:13:07.316 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-20T09:13:07.354 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m0777 -- /var/run/ceph 2023-10-20T09:13:07.409 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-20T09:13:07.409 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/scratch_devs of=/dev/stdout 2023-10-20T09:13:07.457 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-20T09:13:07.458 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_1 2023-10-20T09:13:07.504 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-20T09:13:07.505 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-20T09:13:07.505 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 888 Links: 1 2023-10-20T09:13:07.505 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-20T09:13:07.505 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-10-20 09:07:51.699837555 +0000 2023-10-20T09:13:07.506 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-10-20 09:07:51.335840807 +0000 2023-10-20T09:13:07.506 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-10-20 09:07:51.335840807 +0000 2023-10-20T09:13:07.506 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2023-10-20T09:13:07.506 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-20T09:13:07.558 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-10-20T09:13:07.559 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-10-20T09:13:07.559 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.00033931 s, 1.5 MB/s 2023-10-20T09:13:07.560 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-20T09:13:07.610 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_2 2023-10-20T09:13:07.656 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-20T09:13:07.657 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-20T09:13:07.657 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 898 Links: 1 2023-10-20T09:13:07.657 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-20T09:13:07.657 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-10-20 09:07:52.327831947 +0000 2023-10-20T09:13:07.657 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-10-20 09:07:51.935835449 +0000 2023-10-20T09:13:07.658 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-10-20 09:07:51.935835449 +0000 2023-10-20T09:13:07.658 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2023-10-20T09:13:07.659 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-20T09:13:07.710 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-10-20T09:13:07.711 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-10-20T09:13:07.711 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000351418 s, 1.5 MB/s 2023-10-20T09:13:07.712 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-20T09:13:07.762 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_3 2023-10-20T09:13:07.808 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-20T09:13:07.808 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-20T09:13:07.809 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 921 Links: 1 2023-10-20T09:13:07.809 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-20T09:13:07.809 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-10-20 09:07:52.927826588 +0000 2023-10-20T09:13:07.809 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-10-20 09:07:52.579829695 +0000 2023-10-20T09:13:07.809 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-10-20 09:07:52.579829695 +0000 2023-10-20T09:13:07.810 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2023-10-20T09:13:07.810 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-20T09:13:07.862 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-10-20T09:13:07.863 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-10-20T09:13:07.863 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000384737 s, 1.3 MB/s 2023-10-20T09:13:07.864 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-20T09:13:07.914 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_4 2023-10-20T09:13:07.960 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-20T09:13:07.961 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-20T09:13:07.961 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 938 Links: 1 2023-10-20T09:13:07.961 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-20T09:13:07.961 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-10-20 09:07:53.551821014 +0000 2023-10-20T09:13:07.961 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-10-20 09:07:53.187824265 +0000 2023-10-20T09:13:07.962 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-10-20 09:07:53.187824265 +0000 2023-10-20T09:13:07.962 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2023-10-20T09:13:07.962 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-20T09:13:08.014 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-10-20T09:13:08.015 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-10-20T09:13:08.015 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000292966 s, 1.7 MB/s 2023-10-20T09:13:08.016 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-20T09:13:08.066 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-20T09:13:08.066 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi150.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-20T09:13:08.066 INFO:tasks.ceph:Generating config... 2023-10-20T09:13:08.068 INFO:tasks.ceph:[client] rbd default data pool = datapool 2023-10-20T09:13:08.068 INFO:tasks.ceph:[client] rbd default features = 1 2023-10-20T09:13:08.068 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-10-20T09:13:08.068 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-10-20T09:13:08.069 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-10-20T09:13:08.069 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-20T09:13:08.069 INFO:tasks.ceph:[mgr] debug monc = 20 2023-10-20T09:13:08.070 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-20T09:13:08.070 INFO:tasks.ceph:[mgr] debug objecter = 20 2023-10-20T09:13:08.070 INFO:tasks.ceph:[mgr] debug rados = 20 2023-10-20T09:13:08.071 INFO:tasks.ceph:[mgr] debug rbd = 20 2023-10-20T09:13:08.071 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-20T09:13:08.071 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-20T09:13:08.071 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-20T09:13:08.072 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2023-10-20T09:13:08.072 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2023-10-20T09:13:08.072 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-20T09:13:08.072 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-20T09:13:08.073 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-20T09:13:08.073 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-20T09:13:08.073 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-20T09:13:08.074 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-20T09:13:08.074 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-20T09:13:08.074 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-20T09:13:08.075 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-20T09:13:08.075 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-20T09:13:08.075 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-20T09:13:08.075 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-20T09:13:08.076 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-20T09:13:08.076 INFO:tasks.ceph:Setting up mon.a... 2023-10-20T09:13:08.076 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-20T09:13:08.139 INFO:teuthology.orchestra.run.smithi150.stdout:creating /etc/ceph/ceph.keyring 2023-10-20T09:13:08.156 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-20T09:13:08.234 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-20T09:13:08.291 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.150')] 2023-10-20T09:13:08.292 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.150', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 20, 'debug objecter': 20, 'debug rados': 20, 'debug rbd': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd default data pool': 'datapool', 'rbd default features': 1}, 'mon.a': {}} 2023-10-20T09:13:08.293 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-20T09:13:08.293 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-20T09:13:08.340 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.150 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-20T09:13:08.419 INFO:teuthology.orchestra.run.smithi150.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-20T09:13:08.420 INFO:teuthology.orchestra.run.smithi150.stdout:monmaptool: generated fsid ab6e8fb1-5f4f-4d44-a016-fff308aa08d4 2023-10-20T09:13:08.420 INFO:teuthology.orchestra.run.smithi150.stdout:setting min_mon_release = pacific 2023-10-20T09:13:08.420 INFO:teuthology.orchestra.run.smithi150.stdout:epoch 0 2023-10-20T09:13:08.421 INFO:teuthology.orchestra.run.smithi150.stdout:fsid ab6e8fb1-5f4f-4d44-a016-fff308aa08d4 2023-10-20T09:13:08.421 INFO:teuthology.orchestra.run.smithi150.stdout:last_changed 2023-10-20T09:13:08.420646+0000 2023-10-20T09:13:08.421 INFO:teuthology.orchestra.run.smithi150.stdout:created 2023-10-20T09:13:08.420646+0000 2023-10-20T09:13:08.421 INFO:teuthology.orchestra.run.smithi150.stdout:min_mon_release 16 (pacific) 2023-10-20T09:13:08.421 INFO:teuthology.orchestra.run.smithi150.stdout:election_strategy: 1 2023-10-20T09:13:08.421 INFO:teuthology.orchestra.run.smithi150.stdout:0: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.a 2023-10-20T09:13:08.422 INFO:teuthology.orchestra.run.smithi150.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-20T09:13:08.423 DEBUG:teuthology.orchestra.run.smithi150:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-20T09:13:08.469 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ab6e8fb1-5f4f-4d44-a016-fff308aa08d4... 2023-10-20T09:13:08.470 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-20T09:13:08.542 INFO:teuthology.orchestra.run.smithi150.stdout:[global] 2023-10-20T09:13:08.543 INFO:teuthology.orchestra.run.smithi150.stdout: chdir = "" 2023-10-20T09:13:08.543 INFO:teuthology.orchestra.run.smithi150.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-20T09:13:08.543 INFO:teuthology.orchestra.run.smithi150.stdout: auth supported = cephx 2023-10-20T09:13:08.544 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.544 INFO:teuthology.orchestra.run.smithi150.stdout: filestore xattr use omap = true 2023-10-20T09:13:08.544 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.544 INFO:teuthology.orchestra.run.smithi150.stdout: mon clock drift allowed = 1.000 2023-10-20T09:13:08.544 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.545 INFO:teuthology.orchestra.run.smithi150.stdout: osd crush chooseleaf type = 0 2023-10-20T09:13:08.545 INFO:teuthology.orchestra.run.smithi150.stdout: auth debug = true 2023-10-20T09:13:08.545 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.545 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on old message = true 2023-10-20T09:13:08.545 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on bug = true 2023-10-20T09:13:08.546 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.546 INFO:teuthology.orchestra.run.smithi150.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-20T09:13:08.546 INFO:teuthology.orchestra.run.smithi150.stdout: mon pg warn max object skew = 0 2023-10-20T09:13:08.546 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.546 INFO:teuthology.orchestra.run.smithi150.stdout: # disable pg_autoscaler by default for new pools 2023-10-20T09:13:08.547 INFO:teuthology.orchestra.run.smithi150.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-20T09:13:08.547 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.547 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default size = 2 2023-10-20T09:13:08.547 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.547 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow primary affinity = true 2023-10-20T09:13:08.548 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow pg remap = true 2023-10-20T09:13:08.548 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on legacy crush tunables = false 2023-10-20T09:13:08.548 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on crush straw calc version zero = false 2023-10-20T09:13:08.548 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on no sortbitwise = false 2023-10-20T09:13:08.549 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on osd down out interval zero = false 2023-10-20T09:13:08.549 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on too few osds = false 2023-10-20T09:13:08.550 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-20T09:13:08.550 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-20T09:13:08.550 INFO:teuthology.orchestra.run.smithi150.stdout: mon_allow_pool_size_one = true 2023-10-20T09:13:08.551 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.551 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-20T09:13:08.551 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.552 INFO:teuthology.orchestra.run.smithi150.stdout: osd default data pool replay window = 5 2023-10-20T09:13:08.552 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.553 INFO:teuthology.orchestra.run.smithi150.stdout: mon allow pool delete = true 2023-10-20T09:13:08.553 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.553 INFO:teuthology.orchestra.run.smithi150.stdout: mon cluster log file level = debug 2023-10-20T09:13:08.554 INFO:teuthology.orchestra.run.smithi150.stdout: debug asserts on shutdown = true 2023-10-20T09:13:08.554 INFO:teuthology.orchestra.run.smithi150.stdout: mon health detail to clog = false 2023-10-20T09:13:08.554 INFO:teuthology.orchestra.run.smithi150.stdout: mon host = 172.21.15.150 2023-10-20T09:13:08.554 INFO:teuthology.orchestra.run.smithi150.stdout: mon client directed command retry = 5 2023-10-20T09:13:08.555 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on pool no app = False 2023-10-20T09:13:08.555 INFO:teuthology.orchestra.run.smithi150.stdout: ms inject socket failures = 5000 2023-10-20T09:13:08.555 INFO:teuthology.orchestra.run.smithi150.stdout: fsid = ab6e8fb1-5f4f-4d44-a016-fff308aa08d4 2023-10-20T09:13:08.556 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.556 INFO:teuthology.orchestra.run.smithi150.stdout:[osd] 2023-10-20T09:13:08.556 INFO:teuthology.orchestra.run.smithi150.stdout: osd journal size = 100 2023-10-20T09:13:08.556 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.556 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub load threshold = 5.0 2023-10-20T09:13:08.557 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub max interval = 600 2023-10-20T09:13:08.557 INFO:teuthology.orchestra.run.smithi150.stdout: osd mclock profile = high_recovery_ops 2023-10-20T09:13:08.557 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.557 INFO:teuthology.orchestra.run.smithi150.stdout: osd recover clone overlap = true 2023-10-20T09:13:08.557 INFO:teuthology.orchestra.run.smithi150.stdout: osd recovery max chunk = 1048576 2023-10-20T09:13:08.558 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.558 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug shutdown = true 2023-10-20T09:13:08.558 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug op order = true 2023-10-20T09:13:08.558 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug verify stray on activate = true 2023-10-20T09:13:08.559 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.559 INFO:teuthology.orchestra.run.smithi150.stdout: osd open classes on start = true 2023-10-20T09:13:08.559 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug pg log writeout = true 2023-10-20T09:13:08.559 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.559 INFO:teuthology.orchestra.run.smithi150.stdout: osd deep scrub update digest min age = 30 2023-10-20T09:13:08.560 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.560 INFO:teuthology.orchestra.run.smithi150.stdout: osd map max advance = 10 2023-10-20T09:13:08.560 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.561 INFO:teuthology.orchestra.run.smithi150.stdout: journal zero on create = true 2023-10-20T09:13:08.561 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.561 INFO:teuthology.orchestra.run.smithi150.stdout: filestore ondisk finisher threads = 3 2023-10-20T09:13:08.561 INFO:teuthology.orchestra.run.smithi150.stdout: filestore apply finisher threads = 3 2023-10-20T09:13:08.561 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.561 INFO:teuthology.orchestra.run.smithi150.stdout: bdev debug aio = true 2023-10-20T09:13:08.562 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug misdirected ops = true 2023-10-20T09:13:08.562 INFO:teuthology.orchestra.run.smithi150.stdout: bluefs allocator = hybrid 2023-10-20T09:13:08.562 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore allocator = hybrid 2023-10-20T09:13:08.562 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore block size = 96636764160 2023-10-20T09:13:08.562 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore fsck on mount = True 2023-10-20T09:13:08.563 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluefs = 1/20 2023-10-20T09:13:08.563 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluestore = 1/20 2023-10-20T09:13:08.563 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2023-10-20T09:13:08.563 INFO:teuthology.orchestra.run.smithi150.stdout: debug osd = 20 2023-10-20T09:13:08.563 INFO:teuthology.orchestra.run.smithi150.stdout: debug rocksdb = 4/10 2023-10-20T09:13:08.564 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd backfillfull_ratio = 0.85 2023-10-20T09:13:08.564 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd full ratio = 0.9 2023-10-20T09:13:08.564 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd nearfull ratio = 0.8 2023-10-20T09:13:08.564 INFO:teuthology.orchestra.run.smithi150.stdout: osd failsafe full ratio = 0.95 2023-10-20T09:13:08.564 INFO:teuthology.orchestra.run.smithi150.stdout: osd objectstore = bluestore 2023-10-20T09:13:08.565 INFO:teuthology.orchestra.run.smithi150.stdout: osd shutdown pgref assert = True 2023-10-20T09:13:08.565 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.565 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr] 2023-10-20T09:13:08.565 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2023-10-20T09:13:08.565 INFO:teuthology.orchestra.run.smithi150.stdout: debug mgr = 20 2023-10-20T09:13:08.566 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2023-10-20T09:13:08.566 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2023-10-20T09:13:08.566 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2023-10-20T09:13:08.566 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2023-10-20T09:13:08.566 INFO:teuthology.orchestra.run.smithi150.stdout: mgr/telemetry/nag = false 2023-10-20T09:13:08.567 INFO:teuthology.orchestra.run.smithi150.stdout: debug monc = 20 2023-10-20T09:13:08.567 INFO:teuthology.orchestra.run.smithi150.stdout: debug objecter = 20 2023-10-20T09:13:08.567 INFO:teuthology.orchestra.run.smithi150.stdout: debug rados = 20 2023-10-20T09:13:08.567 INFO:teuthology.orchestra.run.smithi150.stdout: debug rbd = 20 2023-10-20T09:13:08.567 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.568 INFO:teuthology.orchestra.run.smithi150.stdout:[mon] 2023-10-20T09:13:08.568 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2023-10-20T09:13:08.568 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2023-10-20T09:13:08.568 INFO:teuthology.orchestra.run.smithi150.stdout: debug paxos = 20 2023-10-20T09:13:08.568 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2023-10-20T09:13:08.569 INFO:teuthology.orchestra.run.smithi150.stdout: mon data avail warn = 5 2023-10-20T09:13:08.569 INFO:teuthology.orchestra.run.smithi150.stdout: mon mgr mkfs grace = 240 2023-10-20T09:13:08.569 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2023-10-20T09:13:08.569 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd reporter subtree level = osd 2023-10-20T09:13:08.569 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd prime pg temp = true 2023-10-20T09:13:08.570 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2023-10-20T09:13:08.570 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.570 INFO:teuthology.orchestra.run.smithi150.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-20T09:13:08.571 INFO:teuthology.orchestra.run.smithi150.stdout: auth mon ticket ttl = 660 # 11m 2023-10-20T09:13:08.571 INFO:teuthology.orchestra.run.smithi150.stdout: auth service ticket ttl = 240 # 4m 2023-10-20T09:13:08.571 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.571 INFO:teuthology.orchestra.run.smithi150.stdout: # don't complain about insecure global_id in the test suite 2023-10-20T09:13:08.572 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-20T09:13:08.572 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-20T09:13:08.572 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.572 INFO:teuthology.orchestra.run.smithi150.stdout: # 1m isn't quite enough 2023-10-20T09:13:08.572 INFO:teuthology.orchestra.run.smithi150.stdout: mon_down_mkfs_grace = 2m 2023-10-20T09:13:08.572 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.573 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_filestore_osds = false 2023-10-20T09:13:08.573 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:08.573 INFO:teuthology.orchestra.run.smithi150.stdout:[client] 2023-10-20T09:13:08.573 INFO:teuthology.orchestra.run.smithi150.stdout: rgw cache enabled = true 2023-10-20T09:13:08.573 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable ops log = true 2023-10-20T09:13:08.574 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable usage log = true 2023-10-20T09:13:08.574 INFO:teuthology.orchestra.run.smithi150.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-20T09:13:08.574 INFO:teuthology.orchestra.run.smithi150.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-20T09:13:08.574 INFO:teuthology.orchestra.run.smithi150.stdout: rbd default data pool = datapool 2023-10-20T09:13:08.574 INFO:teuthology.orchestra.run.smithi150.stdout: rbd default features = 1 2023-10-20T09:13:08.575 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.a] 2023-10-20T09:13:08.576 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-20T09:13:08.576 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-20T09:13:08.668 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-20T09:13:08.669 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-20T09:13:08.669 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-20T09:13:08.675 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-20T09:13:08.675 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-20T09:13:08.721 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi150.front.sepia.ceph.com 2023-10-20T09:13:08.721 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-20T09:13:08.721 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-20T09:13:08.722 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-20T09:13:08.784 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-20T09:13:08.784 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-20T09:13:08.833 INFO:tasks.ceph:Setting up mon nodes... 2023-10-20T09:13:08.833 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-20T09:13:08.834 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-20T09:13:08.917 INFO:teuthology.orchestra.run.smithi150.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-20T09:13:08.920 INFO:tasks.ceph:Setting up mds nodes... 2023-10-20T09:13:08.920 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-20T09:13:08.921 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-20T09:13:08.996 INFO:teuthology.orchestra.run.smithi150.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-20T09:13:09.006 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-20T09:13:09.006 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi150.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-20T09:13:09.007 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-20T09:13:09.059 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-20T09:13:09.059 INFO:tasks.ceph:role: osd.0 2023-10-20T09:13:09.060 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi150.front.sepia.ceph.com 2023-10-20T09:13:09.060 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-20T09:13:09.120 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-20T09:13:09.121 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-20T09:13:09.121 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-20T09:13:09.121 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-20T09:13:09.121 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-20T09:13:09.121 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2023-10-20T09:13:09.122 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-20T09:13:09.122 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-20T09:13:09.122 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-20T09:13:09.122 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-20T09:13:09.579 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2023-10-20T09:13:09.582 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2023-10-20T09:13:09.582 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-20T09:13:09.674 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-20T09:13:09.685 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: /sbin/restorecon: command not found 2023-10-20T09:13:09.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T09:13:09.687 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-20T09:13:09.739 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-20T09:13:09.739 INFO:tasks.ceph:role: osd.1 2023-10-20T09:13:09.740 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi150.front.sepia.ceph.com 2023-10-20T09:13:09.740 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-20T09:13:09.802 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-20T09:13:09.802 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-20T09:13:09.803 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-20T09:13:09.803 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-20T09:13:09.803 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-20T09:13:09.803 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2023-10-20T09:13:09.804 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-20T09:13:09.804 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-20T09:13:09.804 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-20T09:13:09.804 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-20T09:13:10.268 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2023-10-20T09:13:10.270 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2023-10-20T09:13:10.270 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-20T09:13:10.424 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-20T09:13:10.435 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: /sbin/restorecon: command not found 2023-10-20T09:13:10.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T09:13:10.436 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-20T09:13:10.498 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-20T09:13:10.498 INFO:tasks.ceph:role: osd.2 2023-10-20T09:13:10.498 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi150.front.sepia.ceph.com 2023-10-20T09:13:10.499 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-20T09:13:10.562 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-20T09:13:10.562 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-20T09:13:10.562 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-20T09:13:10.562 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-20T09:13:10.563 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-20T09:13:10.563 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2023-10-20T09:13:10.563 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-20T09:13:10.563 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-20T09:13:10.563 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-20T09:13:10.564 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-20T09:13:11.052 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2023-10-20T09:13:11.054 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2023-10-20T09:13:11.055 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-20T09:13:11.200 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-20T09:13:11.211 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: /sbin/restorecon: command not found 2023-10-20T09:13:11.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T09:13:11.213 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-20T09:13:11.336 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-20T09:13:11.332+0000 7f8e5f8e7640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-20T09:13:11.337 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-20T09:13:11.336+0000 7f8e5f8e7640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-20T09:13:11.337 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-20T09:13:11.336+0000 7f8e5f8e7640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-20T09:13:11.337 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-20T09:13:11.336+0000 7f8e5f8e7640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-20T09:13:11.337 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-20T09:13:11.336+0000 7f8e5f8e7640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-20T09:13:11.338 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-20T09:13:11.336+0000 7f8e5f8e7640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-20T09:13:14.358 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-20T09:13:14.373 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-20T09:13:14.457 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-20T09:13:14.456+0000 7f39d9315640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-20T09:13:14.458 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-20T09:13:14.456+0000 7f39d9315640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-20T09:13:14.458 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-20T09:13:14.456+0000 7f39d9315640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-20T09:13:14.458 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-20T09:13:14.456+0000 7f39d9315640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-20T09:13:14.458 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-20T09:13:14.456+0000 7f39d9315640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-20T09:13:14.459 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-20T09:13:14.456+0000 7f39d9315640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-20T09:13:17.456 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-20T09:13:17.512 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-20T09:13:17.596 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-20T09:13:17.596+0000 7ffb36456640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-20T09:13:17.597 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-20T09:13:17.596+0000 7ffb36456640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-20T09:13:17.597 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-20T09:13:17.596+0000 7ffb36456640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-20T09:13:17.597 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-20T09:13:17.596+0000 7ffb36456640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-20T09:13:17.598 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-20T09:13:17.596+0000 7ffb36456640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-20T09:13:17.598 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-20T09:13:17.596+0000 7ffb36456640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-20T09:13:20.619 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-20T09:13:20.673 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-20T09:13:20.674 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-20T09:13:20.674 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-20T09:13:20.727 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-20T09:13:20.727 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-20T09:13:20.779 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-20T09:13:20.779 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-20T09:13:20.831 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-20T09:13:20.831 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-20T09:13:20.883 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-20T09:13:20.883 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-20T09:13:20.929 INFO:tasks.ceph:Adding keys to all mons... 2023-10-20T09:13:20.929 DEBUG:teuthology.orchestra.run.smithi150:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-20T09:13:20.982 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr.x] 2023-10-20T09:13:20.983 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCkRDJlDq7DNhAA7W9ju1AuWjAnLUHyYqFfNg== 2023-10-20T09:13:20.983 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.0] 2023-10-20T09:13:20.983 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCnRDJlf2waFBAAzDC0/91WQFlCVR5d4LDO/Q== 2023-10-20T09:13:20.983 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.1] 2023-10-20T09:13:20.984 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCqRDJliehRGxAAM9JAG8HIwYaMCqwi4EPd7Q== 2023-10-20T09:13:20.984 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.2] 2023-10-20T09:13:20.984 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCtRDJlLRWjIxAAx6BDexSugu+Gb47e2TuXug== 2023-10-20T09:13:20.984 INFO:teuthology.orchestra.run.smithi150.stdout:[client.0] 2023-10-20T09:13:20.985 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCkRDJl7Il2OxAAH/IEC9KjLusttNfWG7UM5A== 2023-10-20T09:13:20.986 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-20T09:13:21.071 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T09:13:21.152 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T09:13:21.235 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T09:13:21.315 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-20T09:13:21.401 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-20T09:13:21.402 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-20T09:13:21.414 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-20T09:13:21.506 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-20T09:13:21.519 DEBUG:teuthology.orchestra.run.smithi150:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-20T09:13:21.563 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-20T09:13:21.564 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-20T09:13:21.565 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-20T09:13:21.609 INFO:tasks.ceph.mon.a:Started 2023-10-20T09:13:21.610 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-20T09:13:21.610 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-20T09:13:21.611 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-20T09:13:21.613 INFO:tasks.ceph.mgr.x:Started 2023-10-20T09:13:21.613 INFO:tasks.ceph:Setting crush tunables to default 2023-10-20T09:13:21.613 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd crush tunables default 2023-10-20T09:13:21.851 INFO:teuthology.orchestra.run.smithi150.stderr:adjusted tunables profile to default 2023-10-20T09:13:21.869 INFO:tasks.ceph:check_enable_crimson: False 2023-10-20T09:13:21.869 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-20T09:13:21.869 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-20T09:13:21.870 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-20T09:13:21.880 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-20T09:13:21.880 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-20T09:13:21.937 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-20T09:13:21.938 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-20T09:13:21.989 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 234f6989-5b92-4cb0-b780-06ab9137710a 0 2023-10-20T09:13:21.994 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:13:21.992+0000 7f2909cd6180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-20T09:13:22.189 INFO:teuthology.orchestra.run.smithi150.stdout:0 2023-10-20T09:13:22.207 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new bb6392bc-9616-4abc-95b2-83d9879ad204 1 2023-10-20T09:13:22.405 INFO:teuthology.orchestra.run.smithi150.stdout:1 2023-10-20T09:13:22.423 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 38363eb3-42d5-4a63-b24a-50666cca82c7 2 2023-10-20T09:13:22.529 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:13:22.528+0000 7f2909cd6180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-20T09:13:22.618 INFO:teuthology.orchestra.run.smithi150.stdout:2 2023-10-20T09:13:22.635 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-20T09:13:22.636 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-20T09:13:22.638 INFO:tasks.ceph.osd.0:Started 2023-10-20T09:13:22.639 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-20T09:13:22.640 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-20T09:13:22.642 INFO:tasks.ceph.osd.1:Started 2023-10-20T09:13:22.642 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-20T09:13:22.642 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-20T09:13:22.644 INFO:tasks.ceph.osd.2:Started 2023-10-20T09:13:22.645 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T09:13:22.725 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:13:22.724+0000 7f2909cd6180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-20T09:13:22.829 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:22.829 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":5,"fsid":"ab6e8fb1-5f4f-4d44-a016-fff308aa08d4","created":"2023-10-20T09:13:21.742700+0000","modified":"2023-10-20T09:13:22.618839+0000","last_up_change":"0.000000","last_in_change":"2023-10-20T09:13:22.618839+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"234f6989-5b92-4cb0-b780-06ab9137710a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6392bc-9616-4abc-95b2-83d9879ad204","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"38363eb3-42d5-4a63-b24a-50666cca82c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T09:13:22.845 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-20T09:13:22.845 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-20T09:13:23.147 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-20T09:13:23.249 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:13:23.248+0000 7f382148c640 -1 Falling back to public interface 2023-10-20T09:13:23.250 INFO:tasks.ceph.osd.0.smithi150.stderr:2023-10-20T09:13:23.248+0000 7f66bb530640 -1 Falling back to public interface 2023-10-20T09:13:23.265 INFO:tasks.ceph.osd.2.smithi150.stderr:2023-10-20T09:13:23.264+0000 7f1267c94640 -1 Falling back to public interface 2023-10-20T09:13:23.285 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:13:23.284+0000 7f2909cd6180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-20T09:13:23.327 INFO:teuthology.misc.health.smithi150.stdout: 2023-10-20T09:13:23.327 INFO:teuthology.misc.health.smithi150.stdout:{"epoch":5,"fsid":"ab6e8fb1-5f4f-4d44-a016-fff308aa08d4","created":"2023-10-20T09:13:21.742700+0000","modified":"2023-10-20T09:13:22.618839+0000","last_up_change":"0.000000","last_in_change":"2023-10-20T09:13:22.618839+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"234f6989-5b92-4cb0-b780-06ab9137710a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6392bc-9616-4abc-95b2-83d9879ad204","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"38363eb3-42d5-4a63-b24a-50666cca82c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T09:13:23.341 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:13:23.340+0000 7f2909cd6180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-20T09:13:23.342 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-20T09:13:23.393 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:13:23.392+0000 7f2909cd6180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-20T09:13:23.469 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:13:23.468+0000 7f2909cd6180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-20T09:13:23.520 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:13:23.520+0000 7f2909cd6180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-20T09:13:23.597 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:13:23.596+0000 7f2909cd6180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-20T09:13:23.648 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:13:23.648+0000 7f2909cd6180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-20T09:13:23.787 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:13:23.784+0000 7f2909cd6180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-20T09:13:23.846 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:13:23.844+0000 7f2909cd6180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-20T09:13:24.472 INFO:tasks.ceph.mgr.x.smithi150.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-20T09:13:24.472 INFO:tasks.ceph.mgr.x.smithi150.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-20T09:13:24.472 INFO:tasks.ceph.mgr.x.smithi150.stderr: from numpy import show_config as show_numpy_config 2023-10-20T09:13:24.521 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:13:24.520+0000 7f2909cd6180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-20T09:13:24.668 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:13:24.664+0000 7f2909cd6180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-20T09:13:24.837 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:13:24.836+0000 7f2909cd6180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-20T09:13:24.899 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:13:24.896+0000 7f2909cd6180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-20T09:13:25.021 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:13:25.020+0000 7f2909cd6180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-20T09:13:25.622 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:13:25.620+0000 7f2909cd6180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-20T09:13:25.697 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:13:25.696+0000 7f2909cd6180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-20T09:13:25.880 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:13:25.880+0000 7f2909cd6180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-20T09:13:25.954 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:13:25.952+0000 7f2909cd6180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-20T09:13:26.250 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:13:26.248+0000 7f2909cd6180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-20T09:13:26.313 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:13:26.312+0000 7f2909cd6180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-20T09:13:26.603 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:13:26.600+0000 7f382148c640 -1 osd.1 0 log_to_monitors true 2023-10-20T09:13:26.823 INFO:tasks.ceph.osd.2.smithi150.stderr:2023-10-20T09:13:26.820+0000 7f1267c94640 -1 osd.2 0 log_to_monitors true 2023-10-20T09:13:27.094 INFO:tasks.ceph.osd.0.smithi150.stderr:2023-10-20T09:13:27.092+0000 7f66bb530640 -1 osd.0 0 log_to_monitors true 2023-10-20T09:13:27.400 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:13:27.400+0000 7f28b154a640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-20T09:13:29.645 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-20T09:13:30.072 INFO:teuthology.misc.health.smithi150.stdout: 2023-10-20T09:13:30.072 INFO:teuthology.misc.health.smithi150.stdout:{"epoch":7,"fsid":"ab6e8fb1-5f4f-4d44-a016-fff308aa08d4","created":"2023-10-20T09:13:21.742700+0000","modified":"2023-10-20T09:13:28.599330+0000","last_up_change":"0.000000","last_in_change":"2023-10-20T09:13:22.618839+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"234f6989-5b92-4cb0-b780-06ab9137710a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6392bc-9616-4abc-95b2-83d9879ad204","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"38363eb3-42d5-4a63-b24a-50666cca82c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T09:13:30.098 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-20T09:13:30.558 INFO:tasks.ceph.osd.0.smithi150.stderr:2023-10-20T09:13:30.556+0000 7f66b74d8640 -1 osd.0 0 waiting for initial osdmap 2023-10-20T09:13:30.588 INFO:tasks.ceph.osd.0.smithi150.stderr:2023-10-20T09:13:30.588+0000 7f66af30d640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-20T09:13:30.635 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:13:30.632+0000 7f381dc48640 -1 osd.1 0 waiting for initial osdmap 2023-10-20T09:13:30.648 INFO:tasks.ceph.osd.2.smithi150.stderr:2023-10-20T09:13:30.648+0000 7f1263c3c640 -1 osd.2 0 waiting for initial osdmap 2023-10-20T09:13:30.672 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:13:30.672+0000 7f3815269640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-20T09:13:30.680 INFO:tasks.ceph.osd.2.smithi150.stderr:2023-10-20T09:13:30.680+0000 7f125ba71640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-20T09:13:36.401 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-20T09:13:36.693 INFO:teuthology.misc.health.smithi150.stdout: 2023-10-20T09:13:36.694 INFO:teuthology.misc.health.smithi150.stdout:{"epoch":12,"fsid":"ab6e8fb1-5f4f-4d44-a016-fff308aa08d4","created":"2023-10-20T09:13:21.742700+0000","modified":"2023-10-20T09:13:35.710133+0000","last_up_change":"2023-10-20T09:13:31.560690+0000","last_in_change":"2023-10-20T09:13:22.618839+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T09:13:32.620929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"234f6989-5b92-4cb0-b780-06ab9137710a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":3924958301},{"type":"v1","addr":"172.21.15.150:6807","nonce":3924958301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":3924958301},{"type":"v1","addr":"172.21.15.150:6811","nonce":3924958301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3924958301},{"type":"v1","addr":"172.21.15.150:6815","nonce":3924958301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3924958301},{"type":"v1","addr":"172.21.15.150:6813","nonce":3924958301}]},"public_addr":"172.21.15.150:6807/3924958301","cluster_addr":"172.21.15.150:6811/3924958301","heartbeat_back_addr":"172.21.15.150:6815/3924958301","heartbeat_front_addr":"172.21.15.150:6813/3924958301","state":["exists","up"]},{"osd":1,"uuid":"bb6392bc-9616-4abc-95b2-83d9879ad204","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1576706561},{"type":"v1","addr":"172.21.15.150:6801","nonce":1576706561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1576706561},{"type":"v1","addr":"172.21.15.150:6803","nonce":1576706561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1576706561},{"type":"v1","addr":"172.21.15.150:6810","nonce":1576706561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6805","nonce":1576706561},{"type":"v1","addr":"172.21.15.150:6806","nonce":1576706561}]},"public_addr":"172.21.15.150:6801/1576706561","cluster_addr":"172.21.15.150:6803/1576706561","heartbeat_back_addr":"172.21.15.150:6810/1576706561","heartbeat_front_addr":"172.21.15.150:6806/1576706561","state":["exists","up"]},{"osd":2,"uuid":"38363eb3-42d5-4a63-b24a-50666cca82c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2945223909},{"type":"v1","addr":"172.21.15.150:6817","nonce":2945223909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":2945223909},{"type":"v1","addr":"172.21.15.150:6819","nonce":2945223909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":2945223909},{"type":"v1","addr":"172.21.15.150:6823","nonce":2945223909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2945223909},{"type":"v1","addr":"172.21.15.150:6821","nonce":2945223909}]},"public_addr":"172.21.15.150:6817/2945223909","cluster_addr":"172.21.15.150:6819/2945223909","heartbeat_back_addr":"172.21.15.150:6823/2945223909","heartbeat_front_addr":"172.21.15.150:6821/2945223909","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T09:13:28.131113+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T09:13:27.588549+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T09:13:27.784784+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T09:13:36.710 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-20T09:13:36.710 INFO:tasks.ceph:Creating RBD pool 2023-10-20T09:13:36.710 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-20T09:13:37.716 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'rbd' created 2023-10-20T09:13:37.744 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-20T09:13:38.725 INFO:teuthology.orchestra.run.smithi150.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-20T09:13:38.753 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-20T09:13:38.753 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-20T09:13:38.754 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-20T09:13:38.754 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-20T09:13:38.755 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-20T09:13:39.079 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:39.096 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":5,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3074178017},{"type":"v1","addr":"172.21.15.150:6825","nonce":3074178017}]},"active_addr":"172.21.15.150:6825/3074178017","active_change":"2023-10-20T09:13:26.334499+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":457467232}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":86601365}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":2098915286}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":3834546794}]}]} 2023-10-20T09:13:39.099 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-20T09:13:39.099 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-20T09:13:39.099 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T09:13:39.385 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:39.385 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":14,"fsid":"ab6e8fb1-5f4f-4d44-a016-fff308aa08d4","created":"2023-10-20T09:13:21.742700+0000","modified":"2023-10-20T09:13:38.717841+0000","last_up_change":"2023-10-20T09:13:31.560690+0000","last_in_change":"2023-10-20T09:13:22.618839+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T09:13:32.620929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T09:13:37.030861+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"234f6989-5b92-4cb0-b780-06ab9137710a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":3924958301},{"type":"v1","addr":"172.21.15.150:6807","nonce":3924958301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":3924958301},{"type":"v1","addr":"172.21.15.150:6811","nonce":3924958301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3924958301},{"type":"v1","addr":"172.21.15.150:6815","nonce":3924958301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3924958301},{"type":"v1","addr":"172.21.15.150:6813","nonce":3924958301}]},"public_addr":"172.21.15.150:6807/3924958301","cluster_addr":"172.21.15.150:6811/3924958301","heartbeat_back_addr":"172.21.15.150:6815/3924958301","heartbeat_front_addr":"172.21.15.150:6813/3924958301","state":["exists","up"]},{"osd":1,"uuid":"bb6392bc-9616-4abc-95b2-83d9879ad204","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1576706561},{"type":"v1","addr":"172.21.15.150:6801","nonce":1576706561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1576706561},{"type":"v1","addr":"172.21.15.150:6803","nonce":1576706561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1576706561},{"type":"v1","addr":"172.21.15.150:6810","nonce":1576706561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6805","nonce":1576706561},{"type":"v1","addr":"172.21.15.150:6806","nonce":1576706561}]},"public_addr":"172.21.15.150:6801/1576706561","cluster_addr":"172.21.15.150:6803/1576706561","heartbeat_back_addr":"172.21.15.150:6810/1576706561","heartbeat_front_addr":"172.21.15.150:6806/1576706561","state":["exists","up"]},{"osd":2,"uuid":"38363eb3-42d5-4a63-b24a-50666cca82c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2945223909},{"type":"v1","addr":"172.21.15.150:6817","nonce":2945223909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":2945223909},{"type":"v1","addr":"172.21.15.150:6819","nonce":2945223909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":2945223909},{"type":"v1","addr":"172.21.15.150:6823","nonce":2945223909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2945223909},{"type":"v1","addr":"172.21.15.150:6821","nonce":2945223909}]},"public_addr":"172.21.15.150:6817/2945223909","cluster_addr":"172.21.15.150:6819/2945223909","heartbeat_back_addr":"172.21.15.150:6823/2945223909","heartbeat_front_addr":"172.21.15.150:6821/2945223909","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T09:13:28.131113+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T09:13:27.588549+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T09:13:27.784784+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T09:13:39.404 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-20T09:13:39.404 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T09:13:39.689 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:39.689 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":14,"fsid":"ab6e8fb1-5f4f-4d44-a016-fff308aa08d4","created":"2023-10-20T09:13:21.742700+0000","modified":"2023-10-20T09:13:38.717841+0000","last_up_change":"2023-10-20T09:13:31.560690+0000","last_in_change":"2023-10-20T09:13:22.618839+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T09:13:32.620929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T09:13:37.030861+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"234f6989-5b92-4cb0-b780-06ab9137710a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":3924958301},{"type":"v1","addr":"172.21.15.150:6807","nonce":3924958301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":3924958301},{"type":"v1","addr":"172.21.15.150:6811","nonce":3924958301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3924958301},{"type":"v1","addr":"172.21.15.150:6815","nonce":3924958301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3924958301},{"type":"v1","addr":"172.21.15.150:6813","nonce":3924958301}]},"public_addr":"172.21.15.150:6807/3924958301","cluster_addr":"172.21.15.150:6811/3924958301","heartbeat_back_addr":"172.21.15.150:6815/3924958301","heartbeat_front_addr":"172.21.15.150:6813/3924958301","state":["exists","up"]},{"osd":1,"uuid":"bb6392bc-9616-4abc-95b2-83d9879ad204","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1576706561},{"type":"v1","addr":"172.21.15.150:6801","nonce":1576706561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1576706561},{"type":"v1","addr":"172.21.15.150:6803","nonce":1576706561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1576706561},{"type":"v1","addr":"172.21.15.150:6810","nonce":1576706561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6805","nonce":1576706561},{"type":"v1","addr":"172.21.15.150:6806","nonce":1576706561}]},"public_addr":"172.21.15.150:6801/1576706561","cluster_addr":"172.21.15.150:6803/1576706561","heartbeat_back_addr":"172.21.15.150:6810/1576706561","heartbeat_front_addr":"172.21.15.150:6806/1576706561","state":["exists","up"]},{"osd":2,"uuid":"38363eb3-42d5-4a63-b24a-50666cca82c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2945223909},{"type":"v1","addr":"172.21.15.150:6817","nonce":2945223909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":2945223909},{"type":"v1","addr":"172.21.15.150:6819","nonce":2945223909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":2945223909},{"type":"v1","addr":"172.21.15.150:6823","nonce":2945223909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2945223909},{"type":"v1","addr":"172.21.15.150:6821","nonce":2945223909}]},"public_addr":"172.21.15.150:6817/2945223909","cluster_addr":"172.21.15.150:6819/2945223909","heartbeat_back_addr":"172.21.15.150:6823/2945223909","heartbeat_front_addr":"172.21.15.150:6821/2945223909","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T09:13:28.131113+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T09:13:27.588549+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T09:13:27.784784+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T09:13:39.707 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-20T09:13:39.708 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-20T09:13:39.708 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-20T09:13:39.861 INFO:teuthology.orchestra.run.smithi150.stdout:34359738371 2023-10-20T09:13:39.861 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-20T09:13:39.876 INFO:teuthology.orchestra.run.smithi150.stdout:34359738372 2023-10-20T09:13:39.876 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-20T09:13:39.879 INFO:teuthology.orchestra.run.smithi150.stdout:34359738374 2023-10-20T09:13:39.880 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-20T09:13:40.173 INFO:teuthology.orchestra.run.smithi150.stdout:34359738370 2023-10-20T09:13:40.191 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.2 2023-10-20T09:13:40.209 INFO:teuthology.orchestra.run.smithi150.stdout:34359738371 2023-10-20T09:13:40.227 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.1 2023-10-20T09:13:40.230 INFO:teuthology.orchestra.run.smithi150.stdout:34359738373 2023-10-20T09:13:40.247 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738373 for osd.0 2023-10-20T09:13:41.192 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-20T09:13:41.229 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-20T09:13:41.248 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-20T09:13:41.521 INFO:teuthology.orchestra.run.smithi150.stdout:34359738371 2023-10-20T09:13:41.538 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738371 for osd.2 2023-10-20T09:13:41.538 DEBUG:teuthology.parallel:result is None 2023-10-20T09:13:41.561 INFO:teuthology.orchestra.run.smithi150.stdout:34359738372 2023-10-20T09:13:41.563 INFO:teuthology.orchestra.run.smithi150.stdout:34359738374 2023-10-20T09:13:41.580 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.1 2023-10-20T09:13:41.580 DEBUG:teuthology.parallel:result is None 2023-10-20T09:13:41.582 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.0 2023-10-20T09:13:41.583 DEBUG:teuthology.parallel:result is None 2023-10-20T09:13:41.583 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-20T09:13:41.583 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T09:13:41.896 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:41.896 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T09:13:41.912 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-10-20T09:13:40.344514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":83040,"kb_used_data":1528,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032480,"statfs":{"total":289910292480,"available":289825259520,"internally_reserved":0,"allocated":1564672,"data_stored":1285374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4779,"internal_metadata":82963797},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.625945"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-20T09:13:38.747719+0000","last_change":"2023-10-20T09:13:38.742871+0000","last_active":"2023-10-20T09:13:38.747719+0000","last_peered":"2023-10-20T09:13:38.747719+0000","last_clean":"2023-10-20T09:13:38.747719+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T09:13:38.747719+0000","last_undegraded":"2023-10-20T09:13:38.747719+0000","last_fullsized":"2023-10-20T09:13:38.747719+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:52:14.505834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-20T09:13:38.747773+0000","last_change":"2023-10-20T09:13:38.742974+0000","last_active":"2023-10-20T09:13:38.747773+0000","last_peered":"2023-10-20T09:13:38.747773+0000","last_clean":"2023-10-20T09:13:38.747773+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T09:13:38.747773+0000","last_undegraded":"2023-10-20T09:13:38.747773+0000","last_fullsized":"2023-10-20T09:13:38.747773+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:57:33.800159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-20T09:13:38.747673+0000","last_change":"2023-10-20T09:13:38.742635+0000","last_active":"2023-10-20T09:13:38.747673+0000","last_peered":"2023-10-20T09:13:38.747673+0000","last_clean":"2023-10-20T09:13:38.747673+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T09:13:38.747673+0000","last_undegraded":"2023-10-20T09:13:38.747673+0000","last_fullsized":"2023-10-20T09:13:38.747673+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:00:11.534813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-20T09:13:38.747709+0000","last_change":"2023-10-20T09:13:38.742777+0000","last_active":"2023-10-20T09:13:38.747709+0000","last_peered":"2023-10-20T09:13:38.747709+0000","last_clean":"2023-10-20T09:13:38.747709+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T09:13:38.747709+0000","last_undegraded":"2023-10-20T09:13:38.747709+0000","last_fullsized":"2023-10-20T09:13:38.747709+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T15:33:19.616785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-20T09:13:38.743420+0000","last_change":"2023-10-20T09:13:38.741487+0000","last_active":"2023-10-20T09:13:38.743420+0000","last_peered":"2023-10-20T09:13:38.743420+0000","last_clean":"2023-10-20T09:13:38.743420+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T09:13:38.743420+0000","last_undegraded":"2023-10-20T09:13:38.743420+0000","last_fullsized":"2023-10-20T09:13:38.743420+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:45:09.675816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-20T09:13:39.736712+0000","last_change":"2023-10-20T09:13:38.740624+0000","last_active":"2023-10-20T09:13:39.736712+0000","last_peered":"2023-10-20T09:13:39.736712+0000","last_clean":"2023-10-20T09:13:39.736712+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T09:13:39.736712+0000","last_undegraded":"2023-10-20T09:13:39.736712+0000","last_fullsized":"2023-10-20T09:13:39.736712+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:21:43.433612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-20T09:13:39.736712+0000","last_change":"2023-10-20T09:13:38.738881+0000","last_active":"2023-10-20T09:13:39.736712+0000","last_peered":"2023-10-20T09:13:39.736712+0000","last_clean":"2023-10-20T09:13:39.736712+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T09:13:39.736712+0000","last_undegraded":"2023-10-20T09:13:39.736712+0000","last_fullsized":"2023-10-20T09:13:39.736712+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:59:24.678918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-20T09:13:38.740572+0000","last_change":"2023-10-20T09:13:38.738881+0000","last_active":"2023-10-20T09:13:38.740572+0000","last_peered":"2023-10-20T09:13:38.740572+0000","last_clean":"2023-10-20T09:13:38.740572+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T09:13:38.740572+0000","last_undegraded":"2023-10-20T09:13:38.740572+0000","last_fullsized":"2023-10-20T09:13:38.740572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:22:35.234447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":140,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-20T09:13:37.722043+0000","last_change":"2023-10-20T09:13:35.016203+0000","last_active":"2023-10-20T09:13:37.722043+0000","last_peered":"2023-10-20T09:13:37.722043+0000","last_clean":"2023-10-20T09:13:37.722043+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T09:13:37.722043+0000","last_undegraded":"2023-10-20T09:13:37.722043+0000","last_fullsized":"2023-10-20T09:13:37.722043+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:33.563853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:33.563853+0000","last_clean_scrub_stamp":"2023-10-20T09:13:33.563853+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:44:23.181197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":37198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27916,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343924,"statfs":{"total":96636764160,"available":96608178176,"internally_reserved":0,"allocated":716800,"data_stored":624088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":716800,"data_stored":624088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T09:13:41.914 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T09:13:42.171 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:42.171 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T09:13:42.187 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-10-20T09:13:40.344514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":83040,"kb_used_data":1528,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032480,"statfs":{"total":289910292480,"available":289825259520,"internally_reserved":0,"allocated":1564672,"data_stored":1285374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4779,"internal_metadata":82963797},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.625945"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-20T09:13:38.747719+0000","last_change":"2023-10-20T09:13:38.742871+0000","last_active":"2023-10-20T09:13:38.747719+0000","last_peered":"2023-10-20T09:13:38.747719+0000","last_clean":"2023-10-20T09:13:38.747719+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T09:13:38.747719+0000","last_undegraded":"2023-10-20T09:13:38.747719+0000","last_fullsized":"2023-10-20T09:13:38.747719+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:52:14.505834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-20T09:13:38.747773+0000","last_change":"2023-10-20T09:13:38.742974+0000","last_active":"2023-10-20T09:13:38.747773+0000","last_peered":"2023-10-20T09:13:38.747773+0000","last_clean":"2023-10-20T09:13:38.747773+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T09:13:38.747773+0000","last_undegraded":"2023-10-20T09:13:38.747773+0000","last_fullsized":"2023-10-20T09:13:38.747773+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:57:33.800159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-20T09:13:38.747673+0000","last_change":"2023-10-20T09:13:38.742635+0000","last_active":"2023-10-20T09:13:38.747673+0000","last_peered":"2023-10-20T09:13:38.747673+0000","last_clean":"2023-10-20T09:13:38.747673+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T09:13:38.747673+0000","last_undegraded":"2023-10-20T09:13:38.747673+0000","last_fullsized":"2023-10-20T09:13:38.747673+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:00:11.534813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-20T09:13:38.747709+0000","last_change":"2023-10-20T09:13:38.742777+0000","last_active":"2023-10-20T09:13:38.747709+0000","last_peered":"2023-10-20T09:13:38.747709+0000","last_clean":"2023-10-20T09:13:38.747709+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T09:13:38.747709+0000","last_undegraded":"2023-10-20T09:13:38.747709+0000","last_fullsized":"2023-10-20T09:13:38.747709+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T15:33:19.616785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-20T09:13:38.743420+0000","last_change":"2023-10-20T09:13:38.741487+0000","last_active":"2023-10-20T09:13:38.743420+0000","last_peered":"2023-10-20T09:13:38.743420+0000","last_clean":"2023-10-20T09:13:38.743420+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T09:13:38.743420+0000","last_undegraded":"2023-10-20T09:13:38.743420+0000","last_fullsized":"2023-10-20T09:13:38.743420+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:45:09.675816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-20T09:13:39.736712+0000","last_change":"2023-10-20T09:13:38.740624+0000","last_active":"2023-10-20T09:13:39.736712+0000","last_peered":"2023-10-20T09:13:39.736712+0000","last_clean":"2023-10-20T09:13:39.736712+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T09:13:39.736712+0000","last_undegraded":"2023-10-20T09:13:39.736712+0000","last_fullsized":"2023-10-20T09:13:39.736712+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:21:43.433612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-20T09:13:39.736712+0000","last_change":"2023-10-20T09:13:38.738881+0000","last_active":"2023-10-20T09:13:39.736712+0000","last_peered":"2023-10-20T09:13:39.736712+0000","last_clean":"2023-10-20T09:13:39.736712+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T09:13:39.736712+0000","last_undegraded":"2023-10-20T09:13:39.736712+0000","last_fullsized":"2023-10-20T09:13:39.736712+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:59:24.678918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-20T09:13:38.740572+0000","last_change":"2023-10-20T09:13:38.738881+0000","last_active":"2023-10-20T09:13:38.740572+0000","last_peered":"2023-10-20T09:13:38.740572+0000","last_clean":"2023-10-20T09:13:38.740572+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T09:13:38.740572+0000","last_undegraded":"2023-10-20T09:13:38.740572+0000","last_fullsized":"2023-10-20T09:13:38.740572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:22:35.234447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":140,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-20T09:13:37.722043+0000","last_change":"2023-10-20T09:13:35.016203+0000","last_active":"2023-10-20T09:13:37.722043+0000","last_peered":"2023-10-20T09:13:37.722043+0000","last_clean":"2023-10-20T09:13:37.722043+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T09:13:37.722043+0000","last_undegraded":"2023-10-20T09:13:37.722043+0000","last_fullsized":"2023-10-20T09:13:37.722043+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:33.563853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:33.563853+0000","last_clean_scrub_stamp":"2023-10-20T09:13:33.563853+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:44:23.181197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":37198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27916,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343924,"statfs":{"total":96636764160,"available":96608178176,"internally_reserved":0,"allocated":716800,"data_stored":624088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":716800,"data_stored":624088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T09:13:42.188 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-20T09:13:42.189 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-20T09:13:42.189 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-20T09:13:42.189 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-20T09:13:42.479 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T09:13:42.479 INFO:teuthology.orchestra.run.smithi150.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-20T09:13:42.497 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-20T09:13:42.497 INFO:teuthology.run_tasks:Running task exec... 2023-10-20T09:13:42.507 INFO:teuthology.task.exec:Executing custom commands... 2023-10-20T09:13:42.507 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi150.front.sepia.ceph.com 2023-10-20T09:13:42.507 DEBUG:teuthology.orchestra.run.smithi150:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create datapool 4' 2023-10-20T09:13:43.752 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'datapool' created 2023-10-20T09:13:43.778 DEBUG:teuthology.orchestra.run.smithi150:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'rbd pool init datapool' 2023-10-20T09:13:46.808 INFO:teuthology.run_tasks:Running task install... 2023-10-20T09:13:46.818 DEBUG:teuthology.task.install:project ceph 2023-10-20T09:13:46.818 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'}} 2023-10-20T09:13:46.818 DEBUG:teuthology.task.install:config {'extra_system_packages': ['fio'], 'flavor': 'default', 'sha1': '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'} 2023-10-20T09:13:46.818 INFO:teuthology.task.install:Using flavor: default 2023-10-20T09:13:46.824 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-20T09:13:46.824 INFO:teuthology.task.install:extra packages: [] 2023-10-20T09:13:46.825 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-key list | grep Ceph 2023-10-20T09:13:46.884 INFO:teuthology.orchestra.run.smithi150.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-20T09:13:46.913 INFO:teuthology.orchestra.run.smithi150.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-20T09:13:46.913 INFO:teuthology.orchestra.run.smithi150.stdout:uid [ unknown] Ceph.com (release key) 2023-10-20T09:13:46.914 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-20T09:13:46.914 INFO:teuthology.task.install.deb:Installing system (non-project) packages: fio on remote deb x86_64 2023-10-20T09:13:46.915 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-20T09:13:47.064 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default/ 2023-10-20T09:13:47.065 INFO:teuthology.task.install.deb:Package version is 18.0.0-6744-g7f103c7c-1jammy 2023-10-20T09:13:47.233 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-20T09:13:47.233 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-20T09:13:47.248 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-get update 2023-10-20T09:13:47.427 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-20T09:13:47.438 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-20T09:13:47.457 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-20T09:13:47.539 INFO:teuthology.orchestra.run.smithi150.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy InRelease 2023-10-20T09:13:47.576 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:5 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy Release 2023-10-20T09:13:47.613 INFO:teuthology.orchestra.run.smithi150.stdout:Ign:6 https://4.chacra.ceph.com/r/ceph/ajarr-wip-extra-debug-62994-oct-19/7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-20T09:13:51.657 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:7 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-20T09:13:52.727 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2023-10-20T09:13:52.742 DEBUG:teuthology.orchestra.run.smithi150:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6744-g7f103c7c-1jammy cephadm=18.0.0-6744-g7f103c7c-1jammy ceph-mds=18.0.0-6744-g7f103c7c-1jammy ceph-mgr=18.0.0-6744-g7f103c7c-1jammy ceph-common=18.0.0-6744-g7f103c7c-1jammy ceph-fuse=18.0.0-6744-g7f103c7c-1jammy ceph-test=18.0.0-6744-g7f103c7c-1jammy ceph-volume=18.0.0-6744-g7f103c7c-1jammy radosgw=18.0.0-6744-g7f103c7c-1jammy python3-rados=18.0.0-6744-g7f103c7c-1jammy python3-rgw=18.0.0-6744-g7f103c7c-1jammy python3-cephfs=18.0.0-6744-g7f103c7c-1jammy python3-rbd=18.0.0-6744-g7f103c7c-1jammy libcephfs2=18.0.0-6744-g7f103c7c-1jammy libcephfs-dev=18.0.0-6744-g7f103c7c-1jammy librados2=18.0.0-6744-g7f103c7c-1jammy librbd1=18.0.0-6744-g7f103c7c-1jammy rbd-fuse=18.0.0-6744-g7f103c7c-1jammy 2023-10-20T09:13:52.793 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2023-10-20T09:13:52.960 INFO:teuthology.orchestra.run.smithi150.stdout:Building dependency tree... 2023-10-20T09:13:52.961 INFO:teuthology.orchestra.run.smithi150.stdout:Reading state information... 2023-10-20T09:13:53.107 INFO:teuthology.orchestra.run.smithi150.stdout:ceph is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-20T09:13:53.107 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-common is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-20T09:13:53.107 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-fuse is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-20T09:13:53.108 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-mds is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-20T09:13:53.108 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-mgr is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-20T09:13:53.108 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-test is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-20T09:13:53.108 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-volume is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-20T09:13:53.108 INFO:teuthology.orchestra.run.smithi150.stdout:cephadm is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-20T09:13:53.109 INFO:teuthology.orchestra.run.smithi150.stdout:libcephfs-dev is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-20T09:13:53.109 INFO:teuthology.orchestra.run.smithi150.stdout:libcephfs2 is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-20T09:13:53.109 INFO:teuthology.orchestra.run.smithi150.stdout:librados2 is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-20T09:13:53.109 INFO:teuthology.orchestra.run.smithi150.stdout:librbd1 is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-20T09:13:53.109 INFO:teuthology.orchestra.run.smithi150.stdout:python3-cephfs is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-20T09:13:53.109 INFO:teuthology.orchestra.run.smithi150.stdout:python3-rados is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-20T09:13:53.110 INFO:teuthology.orchestra.run.smithi150.stdout:python3-rbd is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-20T09:13:53.110 INFO:teuthology.orchestra.run.smithi150.stdout:python3-rgw is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-20T09:13:53.110 INFO:teuthology.orchestra.run.smithi150.stdout:radosgw is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-20T09:13:53.110 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-fuse is already the newest version (18.0.0-6744-g7f103c7c-1jammy). 2023-10-20T09:13:53.110 INFO:teuthology.orchestra.run.smithi150.stdout:The following packages were automatically installed and are no longer required: 2023-10-20T09:13:53.111 INFO:teuthology.orchestra.run.smithi150.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-20T09:13:53.111 INFO:teuthology.orchestra.run.smithi150.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-20T09:13:53.111 INFO:teuthology.orchestra.run.smithi150.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-20T09:13:53.111 INFO:teuthology.orchestra.run.smithi150.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-20T09:13:53.112 INFO:teuthology.orchestra.run.smithi150.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-20T09:13:53.112 INFO:teuthology.orchestra.run.smithi150.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-20T09:13:53.143 INFO:teuthology.orchestra.run.smithi150.stdout:0 upgraded, 0 newly installed, 0 to remove and 257 not upgraded. 2023-10-20T09:13:53.143 INFO:teuthology.orchestra.run.smithi150.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-20T09:13:53.145 DEBUG:teuthology.orchestra.run.smithi150:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install fio 2023-10-20T09:13:53.237 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2023-10-20T09:13:53.403 INFO:teuthology.orchestra.run.smithi150.stdout:Building dependency tree... 2023-10-20T09:13:53.404 INFO:teuthology.orchestra.run.smithi150.stdout:Reading state information... 2023-10-20T09:13:53.548 INFO:teuthology.orchestra.run.smithi150.stdout:The following packages were automatically installed and are no longer required: 2023-10-20T09:13:53.549 INFO:teuthology.orchestra.run.smithi150.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libiscsi7 libpmemobj1 2023-10-20T09:13:53.549 INFO:teuthology.orchestra.run.smithi150.stdout: libpython2-dev libpython2-stdlib libpython2.7 libpython2.7-dev 2023-10-20T09:13:53.549 INFO:teuthology.orchestra.run.smithi150.stdout: libpython2.7-minimal libpython2.7-stdlib linux-image-5.4.0-135-generic 2023-10-20T09:13:53.549 INFO:teuthology.orchestra.run.smithi150.stdout: linux-modules-5.4.0-135-generic python2 python2-dev python2-minimal 2023-10-20T09:13:53.550 INFO:teuthology.orchestra.run.smithi150.stdout: python2.7 python2.7-dev python2.7-minimal 2023-10-20T09:13:53.550 INFO:teuthology.orchestra.run.smithi150.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-20T09:13:53.550 INFO:teuthology.orchestra.run.smithi150.stdout:The following additional packages will be installed: 2023-10-20T09:13:53.551 INFO:teuthology.orchestra.run.smithi150.stdout: libpmemblk1 2023-10-20T09:13:53.551 INFO:teuthology.orchestra.run.smithi150.stdout:Suggested packages: 2023-10-20T09:13:53.552 INFO:teuthology.orchestra.run.smithi150.stdout: gnuplot gfio python-scipy 2023-10-20T09:13:53.575 INFO:teuthology.orchestra.run.smithi150.stdout:The following NEW packages will be installed: 2023-10-20T09:13:53.576 INFO:teuthology.orchestra.run.smithi150.stdout: fio libpmemblk1 2023-10-20T09:13:53.632 INFO:teuthology.orchestra.run.smithi150.stdout:0 upgraded, 2 newly installed, 0 to remove and 257 not upgraded. 2023-10-20T09:13:53.632 INFO:teuthology.orchestra.run.smithi150.stdout:Need to get 4,112 kB of archives. 2023-10-20T09:13:53.633 INFO:teuthology.orchestra.run.smithi150.stdout:After this operation, 7,813 kB of additional disk space will be used. 2023-10-20T09:13:53.633 INFO:teuthology.orchestra.run.smithi150.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libpmemblk1 amd64 1.11.1-3build1 [65.6 kB] 2023-10-20T09:13:53.700 INFO:teuthology.orchestra.run.smithi150.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 fio amd64 3.28-1 [4,047 kB] 2023-10-20T09:13:54.178 INFO:teuthology.orchestra.run.smithi150.stdout:Fetched 4,112 kB in 0s (15.9 MB/s) 2023-10-20T09:13:54.332 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libpmemblk1:amd64. 2023-10-20T09:13:54.430 INFO:teuthology.orchestra.run.smithi150.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 172810 files and directories currently installed.) 2023-10-20T09:13:54.435 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../libpmemblk1_1.11.1-3build1_amd64.deb ... 2023-10-20T09:13:54.522 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libpmemblk1:amd64 (1.11.1-3build1) ... 2023-10-20T09:13:54.867 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package fio. 2023-10-20T09:13:54.895 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../archives/fio_3.28-1_amd64.deb ... 2023-10-20T09:13:54.933 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking fio (3.28-1) ... 2023-10-20T09:13:55.451 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libpmemblk1:amd64 (1.11.1-3build1) ... 2023-10-20T09:13:55.572 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up fio (3.28-1) ... 2023-10-20T09:13:56.184 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-20T09:13:57.226 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-20T09:13:57.704 INFO:teuthology.orchestra.run.smithi150.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-20T09:13:57.707 DEBUG:teuthology.parallel:result is None 2023-10-20T09:13:57.707 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-20T09:13:57.936 DEBUG:teuthology.orchestra.run.smithi150:> dpkg-query -W -f '${Version}' ceph 2023-10-20T09:13:57.956 INFO:teuthology.orchestra.run.smithi150.stdout:18.0.0-6744-g7f103c7c-1jammy 2023-10-20T09:13:57.956 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6744-g7f103c7c-1jammy 2023-10-20T09:13:57.956 INFO:teuthology.task.install:The correct ceph version 18.0.0-6744-g7f103c7c-1jammy is installed. 2023-10-20T09:13:57.957 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2023-10-20T09:13:57.957 INFO:teuthology.run_tasks:Running task workunit... 2023-10-20T09:13:57.972 INFO:tasks.workunit:Pulling workunits from ref 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-20T09:13:57.972 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-20T09:13:57.973 DEBUG:teuthology.orchestra.run.smithi150:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-20T09:13:58.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T09:13:58.005 INFO:teuthology.orchestra.run.smithi150.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-10-20T09:13:58.005 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-10-20T09:13:58.052 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-10-20T09:13:58.053 DEBUG:teuthology.orchestra.run.smithi150:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-10-20T09:13:58.100 INFO:tasks.workunit:timeout=3h 2023-10-20T09:13:58.100 INFO:tasks.workunit:cleanup=True 2023-10-20T09:13:58.101 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7f103c7cad2e64eeb2f3003665c1f2f43a9381ab 2023-10-20T09:13:58.150 INFO:tasks.workunit.client.0.smithi150.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-20T09:14:59.295 INFO:tasks.workunit.client.0.smithi150.stderr:Note: switching to '7f103c7cad2e64eeb2f3003665c1f2f43a9381ab'. 2023-10-20T09:14:59.296 INFO:tasks.workunit.client.0.smithi150.stderr: 2023-10-20T09:14:59.296 INFO:tasks.workunit.client.0.smithi150.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-20T09:14:59.296 INFO:tasks.workunit.client.0.smithi150.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-20T09:14:59.296 INFO:tasks.workunit.client.0.smithi150.stderr:state without impacting any branches by switching back to a branch. 2023-10-20T09:14:59.296 INFO:tasks.workunit.client.0.smithi150.stderr: 2023-10-20T09:14:59.297 INFO:tasks.workunit.client.0.smithi150.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-20T09:14:59.297 INFO:tasks.workunit.client.0.smithi150.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-20T09:14:59.297 INFO:tasks.workunit.client.0.smithi150.stderr: 2023-10-20T09:14:59.297 INFO:tasks.workunit.client.0.smithi150.stderr: git switch -c 2023-10-20T09:14:59.297 INFO:tasks.workunit.client.0.smithi150.stderr: 2023-10-20T09:14:59.298 INFO:tasks.workunit.client.0.smithi150.stderr:Or undo this operation with: 2023-10-20T09:14:59.298 INFO:tasks.workunit.client.0.smithi150.stderr: 2023-10-20T09:14:59.298 INFO:tasks.workunit.client.0.smithi150.stderr: git switch - 2023-10-20T09:14:59.298 INFO:tasks.workunit.client.0.smithi150.stderr: 2023-10-20T09:14:59.298 INFO:tasks.workunit.client.0.smithi150.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-20T09:14:59.299 INFO:tasks.workunit.client.0.smithi150.stderr: 2023-10-20T09:14:59.299 INFO:tasks.workunit.client.0.smithi150.stderr:HEAD is now at 7f103c7cad2 qa/suites/rbd/cli/workloads/rbd_support_module_recovery: more debug 2023-10-20T09:14:59.300 DEBUG:teuthology.orchestra.run.smithi150:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-20T09:14:59.349 INFO:tasks.workunit.client.0.smithi150.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-20T09:14:59.353 INFO:tasks.workunit.client.0.smithi150.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-20T09:14:59.353 INFO:tasks.workunit.client.0.smithi150.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-20T09:15:00.131 INFO:tasks.workunit.client.0.smithi150.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-20T09:15:00.207 INFO:tasks.workunit.client.0.smithi150.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-20T09:15:00.253 INFO:tasks.workunit.client.0.smithi150.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-20T09:15:00.254 INFO:tasks.workunit.client.0.smithi150.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-20T09:15:00.254 INFO:tasks.workunit.client.0.smithi150.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-20T09:15:00.299 INFO:tasks.workunit.client.0.smithi150.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-20T09:15:00.302 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-20T09:15:00.302 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-20T09:15:00.349 INFO:tasks.workunit:Running workunits matching rbd/rbd_support_module_recovery.sh on client.0... 2023-10-20T09:15:00.351 INFO:tasks.workunit:Running workunit rbd/rbd_support_module_recovery.sh... 2023-10-20T09:15:00.351 DEBUG:teuthology.orchestra.run.smithi150:workunit test rbd/rbd_support_module_recovery.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh 2023-10-20T09:15:00.402 INFO:tasks.workunit.client.0.smithi150.stderr:+ POOL=rbd 2023-10-20T09:15:00.402 INFO:tasks.workunit.client.0.smithi150.stderr:+ IMAGE_PREFIX=image 2023-10-20T09:15:00.402 INFO:tasks.workunit.client.0.smithi150.stderr:+ NUM_IMAGES=20 2023-10-20T09:15:00.403 INFO:tasks.workunit.client.0.smithi150.stderr:+ RUN_TIME=3600 2023-10-20T09:15:00.403 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror pool enable rbd image 2023-10-20T09:15:00.449 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror pool peer add rbd dummy 2023-10-20T09:15:00.486 INFO:tasks.workunit.client.0.smithi150.stdout:37b12618-2aa1-4082-b9bd-a397dff0816e 2023-10-20T09:15:00.489 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i = 1 )) 2023-10-20T09:15:00.490 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:00.490 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image1 2023-10-20T09:15:00.536 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror image enable rbd/image1 snapshot 2023-10-20T09:15:00.812 INFO:tasks.workunit.client.0.smithi150.stdout:Mirroring enabled 2023-10-20T09:15:00.824 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 1m 2023-10-20T09:15:00.884 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:00.884 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:00.884 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image2 2023-10-20T09:15:00.931 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror image enable rbd/image2 snapshot 2023-10-20T09:15:01.821 INFO:tasks.workunit.client.0.smithi150.stdout:Mirroring enabled 2023-10-20T09:15:01.833 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image2 1m 2023-10-20T09:15:01.895 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:01.895 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:01.895 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image3 2023-10-20T09:15:01.944 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror image enable rbd/image3 snapshot 2023-10-20T09:15:02.823 INFO:tasks.workunit.client.0.smithi150.stdout:Mirroring enabled 2023-10-20T09:15:02.837 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image3 1m 2023-10-20T09:15:02.901 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:02.902 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:02.902 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image4 2023-10-20T09:15:02.953 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror image enable rbd/image4 snapshot 2023-10-20T09:15:03.832 INFO:tasks.workunit.client.0.smithi150.stdout:Mirroring enabled 2023-10-20T09:15:03.846 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image4 1m 2023-10-20T09:15:03.910 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:03.911 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:03.911 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image5 2023-10-20T09:15:03.958 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror image enable rbd/image5 snapshot 2023-10-20T09:15:04.848 INFO:tasks.workunit.client.0.smithi150.stdout:Mirroring enabled 2023-10-20T09:15:04.862 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image5 1m 2023-10-20T09:15:04.924 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:04.924 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:04.925 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image6 2023-10-20T09:15:04.974 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror image enable rbd/image6 snapshot 2023-10-20T09:15:05.852 INFO:tasks.workunit.client.0.smithi150.stdout:Mirroring enabled 2023-10-20T09:15:05.866 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image6 1m 2023-10-20T09:15:05.929 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:05.929 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:05.929 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image7 2023-10-20T09:15:05.978 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror image enable rbd/image7 snapshot 2023-10-20T09:15:06.855 INFO:tasks.workunit.client.0.smithi150.stdout:Mirroring enabled 2023-10-20T09:15:06.867 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image7 1m 2023-10-20T09:15:06.928 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:06.928 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:06.929 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image8 2023-10-20T09:15:06.975 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror image enable rbd/image8 snapshot 2023-10-20T09:15:07.867 INFO:tasks.workunit.client.0.smithi150.stdout:Mirroring enabled 2023-10-20T09:15:07.880 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image8 1m 2023-10-20T09:15:07.943 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:07.943 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:07.943 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image9 2023-10-20T09:15:07.991 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror image enable rbd/image9 snapshot 2023-10-20T09:15:08.872 INFO:tasks.workunit.client.0.smithi150.stdout:Mirroring enabled 2023-10-20T09:15:08.886 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image9 1m 2023-10-20T09:15:08.948 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:08.948 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:08.949 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image10 2023-10-20T09:15:08.999 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror image enable rbd/image10 snapshot 2023-10-20T09:15:09.885 INFO:tasks.workunit.client.0.smithi150.stdout:Mirroring enabled 2023-10-20T09:15:09.898 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image10 1m 2023-10-20T09:15:09.965 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:09.965 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:09.965 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image11 2023-10-20T09:15:10.020 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror image enable rbd/image11 snapshot 2023-10-20T09:15:10.878 INFO:tasks.workunit.client.0.smithi150.stdout:Mirroring enabled 2023-10-20T09:15:10.891 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image11 1m 2023-10-20T09:15:10.951 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:10.952 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:10.952 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image12 2023-10-20T09:15:10.999 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror image enable rbd/image12 snapshot 2023-10-20T09:15:11.888 INFO:tasks.workunit.client.0.smithi150.stdout:Mirroring enabled 2023-10-20T09:15:11.897 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image12 1m 2023-10-20T09:15:11.959 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:11.959 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:11.959 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image13 2023-10-20T09:15:12.008 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror image enable rbd/image13 snapshot 2023-10-20T09:15:12.899 INFO:tasks.workunit.client.0.smithi150.stdout:Mirroring enabled 2023-10-20T09:15:12.913 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image13 1m 2023-10-20T09:15:12.979 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:12.979 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:12.980 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image14 2023-10-20T09:15:13.031 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror image enable rbd/image14 snapshot 2023-10-20T09:15:13.914 INFO:tasks.workunit.client.0.smithi150.stdout:Mirroring enabled 2023-10-20T09:15:13.927 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image14 1m 2023-10-20T09:15:13.992 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:13.992 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:13.992 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image15 2023-10-20T09:15:14.041 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror image enable rbd/image15 snapshot 2023-10-20T09:15:14.932 INFO:tasks.workunit.client.0.smithi150.stdout:Mirroring enabled 2023-10-20T09:15:14.945 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image15 1m 2023-10-20T09:15:15.006 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:15.006 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:15.007 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image16 2023-10-20T09:15:15.056 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror image enable rbd/image16 snapshot 2023-10-20T09:15:15.940 INFO:tasks.workunit.client.0.smithi150.stdout:Mirroring enabled 2023-10-20T09:15:15.954 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image16 1m 2023-10-20T09:15:16.016 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:16.016 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:16.016 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image17 2023-10-20T09:15:16.063 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror image enable rbd/image17 snapshot 2023-10-20T09:15:16.948 INFO:tasks.workunit.client.0.smithi150.stdout:Mirroring enabled 2023-10-20T09:15:16.958 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image17 1m 2023-10-20T09:15:17.018 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:17.018 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:17.018 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image18 2023-10-20T09:15:17.068 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror image enable rbd/image18 snapshot 2023-10-20T09:15:17.957 INFO:tasks.workunit.client.0.smithi150.stdout:Mirroring enabled 2023-10-20T09:15:17.973 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image18 1m 2023-10-20T09:15:18.032 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:18.033 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:18.033 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image19 2023-10-20T09:15:18.295 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror image enable rbd/image19 snapshot 2023-10-20T09:15:18.972 INFO:tasks.workunit.client.0.smithi150.stdout:Mirroring enabled 2023-10-20T09:15:18.988 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image19 1m 2023-10-20T09:15:19.049 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:19.050 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:19.050 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image20 2023-10-20T09:15:19.098 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror image enable rbd/image20 snapshot 2023-10-20T09:15:19.975 INFO:tasks.workunit.client.0.smithi150.stdout:Mirroring enabled 2023-10-20T09:15:19.990 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image20 1m 2023-10-20T09:15:20.053 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:20.053 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:20.053 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i = 1 )) 2023-10-20T09:15:20.053 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:20.054 INFO:tasks.workunit.client.0.smithi150.stderr:++ sudo rbd device map rbd/image1 2023-10-20T09:15:20.175 INFO:tasks.workunit.client.0.smithi150.stderr:+ DEVS[$i]=/dev/rbd0 2023-10-20T09:15:20.176 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:20.176 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:20.176 INFO:tasks.workunit.client.0.smithi150.stderr:+ fio --name=fiotest --filename=/dev/rbd0 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-20T09:15:20.176 INFO:tasks.workunit.client.0.smithi150.stderr:++ sudo rbd device map rbd/image2 2023-10-20T09:15:20.247 INFO:tasks.workunit.client.0.smithi150.stderr:+ DEVS[$i]=/dev/rbd1 2023-10-20T09:15:20.247 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:20.247 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:20.248 INFO:tasks.workunit.client.0.smithi150.stderr:+ fio --name=fiotest --filename=/dev/rbd1 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-20T09:15:20.248 INFO:tasks.workunit.client.0.smithi150.stderr:++ sudo rbd device map rbd/image3 2023-10-20T09:15:20.318 INFO:tasks.workunit.client.0.smithi150.stderr:+ DEVS[$i]=/dev/rbd2 2023-10-20T09:15:20.319 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:20.319 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:20.319 INFO:tasks.workunit.client.0.smithi150.stderr:+ fio --name=fiotest --filename=/dev/rbd2 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-20T09:15:20.319 INFO:tasks.workunit.client.0.smithi150.stderr:++ sudo rbd device map rbd/image4 2023-10-20T09:15:20.606 INFO:tasks.workunit.client.0.smithi150.stderr:+ DEVS[$i]=/dev/rbd3 2023-10-20T09:15:20.606 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:20.607 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:20.607 INFO:tasks.workunit.client.0.smithi150.stderr:+ fio --name=fiotest --filename=/dev/rbd3 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-20T09:15:20.607 INFO:tasks.workunit.client.0.smithi150.stderr:++ sudo rbd device map rbd/image5 2023-10-20T09:15:20.709 INFO:tasks.workunit.client.0.smithi150.stderr:+ DEVS[$i]=/dev/rbd4 2023-10-20T09:15:20.709 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:20.710 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:20.710 INFO:tasks.workunit.client.0.smithi150.stderr:+ fio --name=fiotest --filename=/dev/rbd4 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-20T09:15:20.710 INFO:tasks.workunit.client.0.smithi150.stderr:++ sudo rbd device map rbd/image6 2023-10-20T09:15:20.808 INFO:tasks.workunit.client.0.smithi150.stderr:+ DEVS[$i]=/dev/rbd5 2023-10-20T09:15:20.809 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:20.809 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:20.809 INFO:tasks.workunit.client.0.smithi150.stderr:++ sudo rbd device map rbd/image7 2023-10-20T09:15:20.809 INFO:tasks.workunit.client.0.smithi150.stderr:+ fio --name=fiotest --filename=/dev/rbd5 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-20T09:15:20.910 INFO:tasks.workunit.client.0.smithi150.stderr:+ DEVS[$i]=/dev/rbd6 2023-10-20T09:15:20.910 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:20.910 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:20.911 INFO:tasks.workunit.client.0.smithi150.stderr:+ fio --name=fiotest --filename=/dev/rbd6 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-20T09:15:20.911 INFO:tasks.workunit.client.0.smithi150.stderr:++ sudo rbd device map rbd/image8 2023-10-20T09:15:21.021 INFO:tasks.workunit.client.0.smithi150.stderr:+ DEVS[$i]=/dev/rbd7 2023-10-20T09:15:21.021 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:21.022 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:21.022 INFO:tasks.workunit.client.0.smithi150.stderr:+ fio --name=fiotest --filename=/dev/rbd7 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-20T09:15:21.022 INFO:tasks.workunit.client.0.smithi150.stderr:++ sudo rbd device map rbd/image9 2023-10-20T09:15:21.144 INFO:tasks.workunit.client.0.smithi150.stderr:+ DEVS[$i]=/dev/rbd8 2023-10-20T09:15:21.145 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:21.145 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:21.145 INFO:tasks.workunit.client.0.smithi150.stderr:+ fio --name=fiotest --filename=/dev/rbd8 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-20T09:15:21.145 INFO:tasks.workunit.client.0.smithi150.stderr:++ sudo rbd device map rbd/image10 2023-10-20T09:15:21.267 INFO:tasks.workunit.client.0.smithi150.stderr:+ DEVS[$i]=/dev/rbd9 2023-10-20T09:15:21.267 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:21.267 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:21.268 INFO:tasks.workunit.client.0.smithi150.stderr:+ fio --name=fiotest --filename=/dev/rbd9 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-20T09:15:21.268 INFO:tasks.workunit.client.0.smithi150.stderr:++ sudo rbd device map rbd/image11 2023-10-20T09:15:21.419 INFO:tasks.workunit.client.0.smithi150.stderr:+ DEVS[$i]=/dev/rbd10 2023-10-20T09:15:21.420 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:21.420 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:21.420 INFO:tasks.workunit.client.0.smithi150.stderr:+ fio --name=fiotest --filename=/dev/rbd10 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-20T09:15:21.420 INFO:tasks.workunit.client.0.smithi150.stderr:++ sudo rbd device map rbd/image12 2023-10-20T09:15:21.738 INFO:tasks.workunit.client.0.smithi150.stderr:+ DEVS[$i]=/dev/rbd11 2023-10-20T09:15:21.739 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:21.739 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:21.739 INFO:tasks.workunit.client.0.smithi150.stderr:+ fio --name=fiotest --filename=/dev/rbd11 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-20T09:15:21.739 INFO:tasks.workunit.client.0.smithi150.stderr:++ sudo rbd device map rbd/image13 2023-10-20T09:15:21.874 INFO:tasks.workunit.client.0.smithi150.stderr:+ DEVS[$i]=/dev/rbd12 2023-10-20T09:15:21.874 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:21.875 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:21.875 INFO:tasks.workunit.client.0.smithi150.stderr:+ fio --name=fiotest --filename=/dev/rbd12 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-20T09:15:21.875 INFO:tasks.workunit.client.0.smithi150.stderr:++ sudo rbd device map rbd/image14 2023-10-20T09:15:21.980 INFO:tasks.workunit.client.0.smithi150.stderr:+ DEVS[$i]=/dev/rbd13 2023-10-20T09:15:21.981 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:21.981 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:21.981 INFO:tasks.workunit.client.0.smithi150.stderr:+ fio --name=fiotest --filename=/dev/rbd13 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-20T09:15:21.981 INFO:tasks.workunit.client.0.smithi150.stderr:++ sudo rbd device map rbd/image15 2023-10-20T09:15:22.131 INFO:tasks.workunit.client.0.smithi150.stderr:+ DEVS[$i]=/dev/rbd14 2023-10-20T09:15:22.131 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:22.132 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:22.132 INFO:tasks.workunit.client.0.smithi150.stderr:++ sudo rbd device map rbd/image16 2023-10-20T09:15:22.132 INFO:tasks.workunit.client.0.smithi150.stderr:+ fio --name=fiotest --filename=/dev/rbd14 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-20T09:15:22.258 INFO:tasks.workunit.client.0.smithi150.stderr:+ DEVS[$i]=/dev/rbd15 2023-10-20T09:15:22.258 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:22.259 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:22.259 INFO:tasks.workunit.client.0.smithi150.stderr:+ fio --name=fiotest --filename=/dev/rbd15 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-20T09:15:22.259 INFO:tasks.workunit.client.0.smithi150.stderr:++ sudo rbd device map rbd/image17 2023-10-20T09:15:22.392 INFO:tasks.workunit.client.0.smithi150.stderr:+ DEVS[$i]=/dev/rbd16 2023-10-20T09:15:22.392 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:22.392 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:22.392 INFO:tasks.workunit.client.0.smithi150.stderr:+ fio --name=fiotest --filename=/dev/rbd16 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-20T09:15:22.393 INFO:tasks.workunit.client.0.smithi150.stderr:++ sudo rbd device map rbd/image18 2023-10-20T09:15:22.597 INFO:tasks.workunit.client.0.smithi150.stderr:+ DEVS[$i]=/dev/rbd17 2023-10-20T09:15:22.597 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:22.598 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:22.598 INFO:tasks.workunit.client.0.smithi150.stderr:+ fio --name=fiotest --filename=/dev/rbd17 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-20T09:15:22.598 INFO:tasks.workunit.client.0.smithi150.stderr:++ sudo rbd device map rbd/image19 2023-10-20T09:15:22.800 INFO:tasks.workunit.client.0.smithi150.stderr:+ DEVS[$i]=/dev/rbd18 2023-10-20T09:15:22.800 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:22.800 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:22.802 INFO:tasks.workunit.client.0.smithi150.stderr:+ fio --name=fiotest --filename=/dev/rbd18 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-20T09:15:22.803 INFO:tasks.workunit.client.0.smithi150.stderr:++ sudo rbd device map rbd/image20 2023-10-20T09:15:22.982 INFO:tasks.workunit.client.0.smithi150.stderr:+ DEVS[$i]=/dev/rbd19 2023-10-20T09:15:22.982 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T09:15:22.983 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 20 )) 2023-10-20T09:15:22.983 INFO:tasks.workunit.client.0.smithi150.stderr:+ fio --name=fiotest --filename=/dev/rbd19 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-20T09:15:22.985 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:15:22.986 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793322 2023-10-20T09:15:22.986 INFO:tasks.workunit.client.0.smithi150.stderr:+ END_TIME=1697796922 2023-10-20T09:15:22.987 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR= 2023-10-20T09:15:22.987 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR= 2023-10-20T09:15:22.987 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:15:22.988 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n '' ]] 2023-10-20T09:15:22.988 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:15:28.409 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:15:28.404+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T09:15:28.410 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:15:28.404+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:15:28.410 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:15:28.404+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T09:15:28.410 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:15:28.404+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:15:28.410 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:15:28.404+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:15:28.410 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:15:28.404+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:15:32.988 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:15:32.989 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:15:32.989 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:15:32.991 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:15:33.879 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2098915286 2023-10-20T09:15:33.879 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:15:33.880 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793333 2023-10-20T09:15:33.881 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:15:33.881 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2098915286 ]] 2023-10-20T09:15:33.881 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2098915286 != '' ]] 2023-10-20T09:15:33.881 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/2098915286 2023-10-20T09:15:35.740 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/2098915286 until 2023-10-20T10:15:34.984990+0000 (3600 sec) 2023-10-20T09:15:35.769 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:15:35.770 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/2098915286 2023-10-20T09:15:36.465 INFO:tasks.workunit.client.0.smithi150.stderr:listed 1 entries 2023-10-20T09:15:36.493 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/2098915286 2023-10-20T09:15:36.493 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:15:39.926 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:15:39.924+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:15:39.929 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:15:39.924+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:15:46.495 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:15:46.495 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:15:46.496 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:15:46.497 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:15:47.428 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2098915286 2023-10-20T09:15:47.428 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:15:47.429 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793347 2023-10-20T09:15:47.429 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:15:47.429 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2098915286 ]] 2023-10-20T09:15:47.430 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2098915286 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\0\9\8\9\1\5\2\8\6 ]] 2023-10-20T09:15:47.430 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:15:57.431 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:15:57.433 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:15:57.445 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:15:57.451 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:15:58.392 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2098915286 2023-10-20T09:15:58.400 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:15:58.402 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793358 2023-10-20T09:15:58.403 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:15:58.403 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2098915286 ]] 2023-10-20T09:15:58.403 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2098915286 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\0\9\8\9\1\5\2\8\6 ]] 2023-10-20T09:15:58.403 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:16:00.008 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.003+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.009 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.003+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc210e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.013 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.007+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.013 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.007+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bb8f1c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.021 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.015+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.022 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.015+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc073700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.025 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.019+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.025 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.019+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc072000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.035 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.031+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.035 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.031+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc072480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.057 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.051+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.057 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.051+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bbf9dd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.064 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.059+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.065 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.059+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc072180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.081 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.075+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.081 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.075+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc211600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.086 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.083+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.087 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.083+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc211c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.102 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.095+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.102 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.095+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc072180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.103 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.099+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.103 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.099+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc072880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.111 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.103+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.112 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.103+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc072000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.117 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.111+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.117 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.111+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bb8d2800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.126 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.123+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.126 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.123+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bbf9d900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.129 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.123+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.129 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.123+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc072900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.134 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.131+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.135 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.131+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc072d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.145 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.139+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.145 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.139+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc072080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.152 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.147+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.152 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.147+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc211600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.157 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.151+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.157 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.151+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc211c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.164 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.159+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:00.164 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:16:00.159+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bb8f1c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:16:08.404 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:16:08.405 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:16:08.410 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:16:08.416 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:16:09.217 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2098915286 2023-10-20T09:16:09.217 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:16:09.218 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793369 2023-10-20T09:16:09.218 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:16:09.218 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2098915286 ]] 2023-10-20T09:16:09.219 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2098915286 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\0\9\8\9\1\5\2\8\6 ]] 2023-10-20T09:16:09.219 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:16:12.078 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:16:12.070+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:16:12.122 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:16:12.074+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T09:16:12.123 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:16:12.074+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:16:12.123 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:16:12.074+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab4f70 2023-10-20T09:16:12.123 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:16:12.074+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T09:16:12.123 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:16:12.074+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:16:12.124 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:16:12.074+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:16:12.124 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:16:12.074+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T09:16:12.124 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:16:12.074+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:16:12.124 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:16:12.074+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:16:12.124 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:16:12.074+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2750 2023-10-20T09:16:12.124 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:16:12.074+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2820 2023-10-20T09:16:19.220 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:16:19.220 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:16:19.220 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:16:19.222 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:16:20.083 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2098915286 2023-10-20T09:16:20.086 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:16:20.087 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793380 2023-10-20T09:16:20.087 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:16:20.087 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2098915286 ]] 2023-10-20T09:16:20.088 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2098915286 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\0\9\8\9\1\5\2\8\6 ]] 2023-10-20T09:16:20.088 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:16:30.093 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:16:30.093 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:16:30.093 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:16:30.094 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:16:30.972 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2098915286 2023-10-20T09:16:30.972 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:16:30.973 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793390 2023-10-20T09:16:30.973 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:16:30.973 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2098915286 ]] 2023-10-20T09:16:30.973 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2098915286 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\0\9\8\9\1\5\2\8\6 ]] 2023-10-20T09:16:30.974 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:16:40.975 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:16:40.975 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:16:40.976 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:16:40.978 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:16:41.846 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1049815273 2023-10-20T09:16:41.846 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:16:41.847 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793401 2023-10-20T09:16:41.848 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:16:41.848 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1049815273 ]] 2023-10-20T09:16:41.848 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1049815273 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\0\9\8\9\1\5\2\8\6 ]] 2023-10-20T09:16:41.848 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/1049815273 2023-10-20T09:16:43.850 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/1049815273 until 2023-10-20T10:16:43.100186+0000 (3600 sec) 2023-10-20T09:16:43.888 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:16:43.894 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/1049815273 2023-10-20T09:16:44.666 INFO:tasks.workunit.client.0.smithi150.stderr:listed 2 entries 2023-10-20T09:16:44.702 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/1049815273 2023-10-20T09:16:44.702 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:16:54.704 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:16:54.705 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:16:54.705 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:16:54.707 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:16:55.556 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1049815273 2023-10-20T09:16:55.559 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:16:55.560 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793415 2023-10-20T09:16:55.560 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:16:55.560 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1049815273 ]] 2023-10-20T09:16:55.561 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1049815273 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\0\4\9\8\1\5\2\7\3 ]] 2023-10-20T09:16:55.561 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:17:00.009 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.002+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.010 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.002+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.010 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.002+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bbf9cb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.010 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.002+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc211300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.012 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.006+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.012 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.006+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc211d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.017 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.010+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.018 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.010+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bbf9dd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.023 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.018+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.023 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.018+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc210980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.028 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.022+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.029 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.022+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc211c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.034 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.030+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.035 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.030+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bbf9d900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.048 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.042+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.048 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.042+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bb8d2600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.055 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.050+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.056 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.050+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc211a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.060 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.054+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.060 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.054+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc211300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.063 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.058+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.063 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.058+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc211600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.078 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.070+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.078 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.070+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bb8f1800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.090 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.086+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.090 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.086+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bbf9d900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.096 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.090+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.096 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.090+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc210600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.117 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.110+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.117 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.110+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc073600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.122 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.118+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.123 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.118+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc072b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.140 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.134+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.141 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.134+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc210980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.141 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.134+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.141 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.134+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bb8d3100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.156 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.150+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.156 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.150+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bb8d3e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.164 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.158+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:00.164 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:17:00.158+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc211300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:17:05.562 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:17:05.562 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:17:05.563 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:17:05.569 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:17:06.421 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1049815273 2023-10-20T09:17:06.421 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:17:06.422 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793426 2023-10-20T09:17:06.422 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:17:06.423 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1049815273 ]] 2023-10-20T09:17:06.423 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1049815273 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\0\4\9\8\1\5\2\7\3 ]] 2023-10-20T09:17:06.423 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:17:16.436 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:17:16.445 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:17:16.446 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:17:16.449 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:17:16.683 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:17:16.674+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:17:16.684 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:17:16.674+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T09:17:16.684 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:17:16.674+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:17:16.685 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:17:16.674+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab4f70 2023-10-20T09:17:16.685 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:17:16.674+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T09:17:16.685 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:17:16.674+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:17:16.686 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:17:16.674+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:17:16.686 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:17:16.674+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T09:17:16.686 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:17:16.674+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:17:16.686 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:17:16.674+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:17:17.388 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1049815273 2023-10-20T09:17:17.388 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:17:17.389 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793437 2023-10-20T09:17:17.389 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:17:17.389 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1049815273 ]] 2023-10-20T09:17:17.389 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1049815273 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\0\4\9\8\1\5\2\7\3 ]] 2023-10-20T09:17:17.390 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:17:27.181 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:17:27.173+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:17:27.182 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:17:27.173+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T09:17:27.182 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:17:27.173+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:17:27.182 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:17:27.173+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab4f70 2023-10-20T09:17:27.182 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:17:27.173+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T09:17:27.183 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:17:27.173+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:17:27.183 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:17:27.173+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:17:27.183 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:17:27.173+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T09:17:27.183 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:17:27.173+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:17:27.184 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:17:27.173+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:17:27.184 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:17:27.173+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2750 2023-10-20T09:17:27.184 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:17:27.173+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2820 2023-10-20T09:17:27.390 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:17:27.391 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:17:27.391 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:17:27.398 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:17:28.387 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1049815273 2023-10-20T09:17:28.389 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:17:28.390 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793448 2023-10-20T09:17:28.391 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:17:28.391 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1049815273 ]] 2023-10-20T09:17:28.391 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1049815273 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\0\4\9\8\1\5\2\7\3 ]] 2023-10-20T09:17:28.391 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:17:38.402 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:17:38.404 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:17:38.406 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:17:38.407 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:17:39.418 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1049815273 2023-10-20T09:17:39.418 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:17:39.420 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793459 2023-10-20T09:17:39.420 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:17:39.420 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1049815273 ]] 2023-10-20T09:17:39.420 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1049815273 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\0\4\9\8\1\5\2\7\3 ]] 2023-10-20T09:17:39.421 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:17:49.426 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:17:49.429 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:17:49.437 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:17:49.448 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:17:50.356 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/674319495 2023-10-20T09:17:50.359 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:17:50.360 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793470 2023-10-20T09:17:50.361 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:17:50.361 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/674319495 ]] 2023-10-20T09:17:50.361 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/674319495 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\0\4\9\8\1\5\2\7\3 ]] 2023-10-20T09:17:50.361 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/674319495 2023-10-20T09:17:52.852 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/674319495 until 2023-10-20T10:17:52.057124+0000 (3600 sec) 2023-10-20T09:17:52.891 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:17:52.894 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/674319495 2023-10-20T09:17:53.622 INFO:tasks.workunit.client.0.smithi150.stderr:listed 3 entries 2023-10-20T09:17:53.673 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/674319495 2023-10-20T09:17:53.673 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:18:00.008 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.001+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.009 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.001+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc3b2400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.010 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.005+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.010 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.005+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bb990e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.015 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.009+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.015 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.009+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bbf9c780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.019 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.013+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.019 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.013+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bb990680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.022 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.017+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.023 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.017+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bbf9db00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.026 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.021+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.027 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.021+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc3b2400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.031 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.025+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.031 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.025+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bb990c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.035 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.029+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.035 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.029+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc072e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.039 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.033+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.039 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.033+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc3b2280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.044 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.037+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.044 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.037+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bb990680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.047 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.041+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.047 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.041+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bb990100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.051 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.045+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.051 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.045+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bb990500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.055 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.049+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.056 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.049+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bb8d2900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.060 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.053+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.060 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.053+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bb7c7600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.063 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.057+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.063 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.057+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bb7c7980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.067 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.061+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.067 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.061+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc004580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.071 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.065+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.071 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.065+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc004380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.074 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.069+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.074 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.069+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc211180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.079 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.073+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.079 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.073+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc004b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.085 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.081+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:00.085 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:18:00.081+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc004a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:18:03.674 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:18:03.677 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:18:03.679 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:18:03.680 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:18:04.500 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/674319495 2023-10-20T09:18:04.500 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:18:04.501 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793484 2023-10-20T09:18:04.501 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:18:04.502 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/674319495 ]] 2023-10-20T09:18:04.502 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/674319495 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\6\7\4\3\1\9\4\9\5 ]] 2023-10-20T09:18:04.502 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:18:14.510 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:18:14.510 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:18:14.511 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:18:14.515 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:18:15.392 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/674319495 2023-10-20T09:18:15.392 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:18:15.393 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793495 2023-10-20T09:18:15.393 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:18:15.393 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/674319495 ]] 2023-10-20T09:18:15.393 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/674319495 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\6\7\4\3\1\9\4\9\5 ]] 2023-10-20T09:18:15.393 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:18:25.394 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:18:25.395 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:18:25.395 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:18:25.401 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:18:26.264 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/674319495 2023-10-20T09:18:26.265 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:18:26.265 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793506 2023-10-20T09:18:26.266 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:18:26.266 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/674319495 ]] 2023-10-20T09:18:26.266 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/674319495 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\6\7\4\3\1\9\4\9\5 ]] 2023-10-20T09:18:26.266 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:18:36.267 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:18:36.267 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:18:36.278 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:18:36.280 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:18:37.291 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/674319495 2023-10-20T09:18:37.291 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:18:37.292 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793517 2023-10-20T09:18:37.292 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:18:37.292 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/674319495 ]] 2023-10-20T09:18:37.293 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/674319495 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\6\7\4\3\1\9\4\9\5 ]] 2023-10-20T09:18:37.293 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:18:47.294 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:18:47.294 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:18:47.295 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:18:47.296 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:18:48.230 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/674319495 2023-10-20T09:18:48.231 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:18:48.231 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793528 2023-10-20T09:18:48.231 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:18:48.231 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/674319495 ]] 2023-10-20T09:18:48.232 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/674319495 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\6\7\4\3\1\9\4\9\5 ]] 2023-10-20T09:18:48.232 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:18:58.233 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:18:58.233 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:18:58.237 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:18:58.240 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:18:59.139 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1900394575 2023-10-20T09:18:59.139 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:18:59.139 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793539 2023-10-20T09:18:59.140 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:18:59.140 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1900394575 ]] 2023-10-20T09:18:59.140 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1900394575 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\6\7\4\3\1\9\4\9\5 ]] 2023-10-20T09:18:59.140 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/1900394575 2023-10-20T09:19:00.006 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.000+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.006 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.000+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc6b0e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.007 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.004+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.007 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.004+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc72a580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.011 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.008+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.011 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.008+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc72a180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.013 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.012+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.014 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.012+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc72b100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.023 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.020+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.023 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.020+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc72b300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.027 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.024+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.027 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.024+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc6b1d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.044 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.040+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.044 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.040+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc72af00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.069 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.064+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.069 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.064+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc6b1e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.079 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.076+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.080 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.076+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc72af80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.084 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.080+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.085 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.084+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc72a680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.092 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.088+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.093 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.088+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc72a300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.102 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.100+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.102 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.100+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bb990880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.110 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.108+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.110 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.108+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc72b100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.118 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.116+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.119 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.116+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc72a600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.126 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.120+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.126 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.120+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc72b600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.131 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.128+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.132 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.128+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc72b300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.140 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.136+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.141 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.136+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc6b1c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.146 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.144+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.147 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.144+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc6b1a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.152 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.148+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.152 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.148+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc72a200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.157 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.156+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.157 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:19:00.156+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc72a000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:19:00.798 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/1900394575 until 2023-10-20T10:18:59.981153+0000 (3600 sec) 2023-10-20T09:19:00.832 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:19:00.834 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/1900394575 2023-10-20T09:19:01.545 INFO:tasks.workunit.client.0.smithi150.stderr:listed 4 entries 2023-10-20T09:19:01.576 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/1900394575 2023-10-20T09:19:01.577 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:19:11.590 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:19:11.592 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:19:11.595 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:19:11.600 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:19:12.562 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1900394575 2023-10-20T09:19:12.562 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:19:12.563 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793552 2023-10-20T09:19:12.563 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:19:12.563 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1900394575 ]] 2023-10-20T09:19:12.563 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1900394575 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\9\0\0\3\9\4\5\7\5 ]] 2023-10-20T09:19:12.563 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:19:22.565 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:19:22.565 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:19:22.565 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:19:22.569 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:19:23.455 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1900394575 2023-10-20T09:19:23.455 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:19:23.456 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793563 2023-10-20T09:19:23.456 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:19:23.456 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1900394575 ]] 2023-10-20T09:19:23.457 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1900394575 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\9\0\0\3\9\4\5\7\5 ]] 2023-10-20T09:19:23.457 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:19:33.457 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:19:33.458 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:19:33.458 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:19:33.459 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:19:34.342 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1900394575 2023-10-20T09:19:34.342 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:19:34.343 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793574 2023-10-20T09:19:34.343 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:19:34.343 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1900394575 ]] 2023-10-20T09:19:34.343 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1900394575 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\9\0\0\3\9\4\5\7\5 ]] 2023-10-20T09:19:34.344 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:19:44.352 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:19:44.352 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:19:44.357 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:19:44.358 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:19:45.356 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1900394575 2023-10-20T09:19:45.356 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:19:45.357 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793585 2023-10-20T09:19:45.357 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:19:45.357 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1900394575 ]] 2023-10-20T09:19:45.357 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1900394575 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\9\0\0\3\9\4\5\7\5 ]] 2023-10-20T09:19:45.357 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:19:55.359 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:19:55.359 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:19:55.360 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:19:55.361 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:19:56.243 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1900394575 2023-10-20T09:19:56.244 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:19:56.244 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793596 2023-10-20T09:19:56.245 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:19:56.245 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1900394575 ]] 2023-10-20T09:19:56.245 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1900394575 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\9\0\0\3\9\4\5\7\5 ]] 2023-10-20T09:19:56.245 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:20:06.254 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:20:06.255 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:20:06.255 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:20:06.259 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:20:07.120 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1900394575 2023-10-20T09:20:07.120 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:20:07.121 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793607 2023-10-20T09:20:07.121 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:20:07.121 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1900394575 ]] 2023-10-20T09:20:07.121 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1900394575 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\9\0\0\3\9\4\5\7\5 ]] 2023-10-20T09:20:07.122 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:20:17.123 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:20:17.124 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:20:17.132 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:20:17.140 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:20:18.028 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/935800402 2023-10-20T09:20:18.028 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:20:18.029 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793618 2023-10-20T09:20:18.029 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:20:18.029 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/935800402 ]] 2023-10-20T09:20:18.029 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/935800402 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\9\0\0\3\9\4\5\7\5 ]] 2023-10-20T09:20:18.030 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/935800402 2023-10-20T09:20:19.316 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:20:19.312+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:20:19.854 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/935800402 until 2023-10-20T10:20:19.106257+0000 (3600 sec) 2023-10-20T09:20:19.920 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:20:19.921 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/935800402 2023-10-20T09:20:20.594 INFO:tasks.workunit.client.0.smithi150.stderr:listed 5 entries 2023-10-20T09:20:20.629 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/935800402 2023-10-20T09:20:20.630 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:20:30.636 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:20:30.636 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:20:30.637 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:20:30.644 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:20:31.561 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/935800402 2023-10-20T09:20:31.561 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:20:31.562 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793631 2023-10-20T09:20:31.562 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:20:31.562 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/935800402 ]] 2023-10-20T09:20:31.563 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/935800402 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\9\3\5\8\0\0\4\0\2 ]] 2023-10-20T09:20:31.563 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:20:41.564 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:20:41.565 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:20:41.584 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:20:41.585 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:20:42.537 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/935800402 2023-10-20T09:20:42.537 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:20:42.538 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793642 2023-10-20T09:20:42.538 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:20:42.538 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/935800402 ]] 2023-10-20T09:20:42.538 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/935800402 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\9\3\5\8\0\0\4\0\2 ]] 2023-10-20T09:20:42.539 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:20:52.551 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:20:52.553 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:20:52.553 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:20:52.555 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:20:53.400 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/935800402 2023-10-20T09:20:53.400 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:20:53.401 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793653 2023-10-20T09:20:53.401 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:20:53.401 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/935800402 ]] 2023-10-20T09:20:53.401 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/935800402 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\9\3\5\8\0\0\4\0\2 ]] 2023-10-20T09:20:53.402 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:21:00.036 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.031+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.036 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.031+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.036 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.031+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc72bf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.037 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.031+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc4ff700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.038 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.035+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.038 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.035+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.038 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.035+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc4ff800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.038 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.035+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc4ffa00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.039 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.035+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.039 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.035+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc4ffc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.047 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.043+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.047 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.043+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc4ffe00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.057 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.055+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.057 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.055+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc920000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.059 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.055+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.060 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.055+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc920200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.062 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.059+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.062 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.059+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc920400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.064 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.063+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.065 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.063+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc920600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.068 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.063+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.068 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.067+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc920800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.074 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.071+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.074 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.071+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc920a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.079 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.075+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.079 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.075+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc920c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.083 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.079+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.083 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.079+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc920e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.103 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.099+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.104 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.099+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc921000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.104 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.099+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.104 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.099+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc921800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.116 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.111+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.116 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.111+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc921680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.123 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.119+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.124 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.119+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc921480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.134 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.131+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.135 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.131+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc921200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.141 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.139+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:00.142 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:00.139+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc921a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:21:03.403 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:21:03.410 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:21:03.411 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:21:03.412 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:21:04.243 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/935800402 2023-10-20T09:21:04.243 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:21:04.244 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793664 2023-10-20T09:21:04.244 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:21:04.245 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/935800402 ]] 2023-10-20T09:21:04.245 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/935800402 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\9\3\5\8\0\0\4\0\2 ]] 2023-10-20T09:21:04.245 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:21:08.112 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:21:08.103+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:21:11.228 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:21:11.219+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:21:14.258 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:21:14.259 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:21:14.259 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:21:14.264 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:21:15.232 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/935800402 2023-10-20T09:21:15.232 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:21:15.233 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793675 2023-10-20T09:21:15.233 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:21:15.233 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/935800402 ]] 2023-10-20T09:21:15.233 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/935800402 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\9\3\5\8\0\0\4\0\2 ]] 2023-10-20T09:21:15.233 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:21:25.234 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:21:25.235 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:21:25.235 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:21:25.245 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:21:26.050 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1998324367 2023-10-20T09:21:26.051 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:21:26.052 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793686 2023-10-20T09:21:26.052 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:21:26.053 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1998324367 ]] 2023-10-20T09:21:26.053 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1998324367 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\9\3\5\8\0\0\4\0\2 ]] 2023-10-20T09:21:26.054 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/1998324367 2023-10-20T09:21:27.896 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/1998324367 until 2023-10-20T10:21:27.122105+0000 (3600 sec) 2023-10-20T09:21:27.940 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/1998324367 2023-10-20T09:21:27.944 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:21:28.681 INFO:tasks.workunit.client.0.smithi150.stderr:listed 6 entries 2023-10-20T09:21:28.708 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/1998324367 2023-10-20T09:21:28.708 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:21:38.709 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:21:38.710 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:21:38.710 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:21:38.710 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:21:39.572 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1998324367 2023-10-20T09:21:39.573 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:21:39.573 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793699 2023-10-20T09:21:39.574 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:21:39.574 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1998324367 ]] 2023-10-20T09:21:39.574 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1998324367 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\9\9\8\3\2\4\3\6\7 ]] 2023-10-20T09:21:39.574 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:21:49.575 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:21:49.576 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:21:49.580 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:21:49.584 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:21:50.672 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1998324367 2023-10-20T09:21:50.672 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:21:50.673 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793710 2023-10-20T09:21:50.673 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:21:50.674 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1998324367 ]] 2023-10-20T09:21:50.674 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1998324367 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\9\9\8\3\2\4\3\6\7 ]] 2023-10-20T09:21:50.674 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:21:53.683 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:21:53.682+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:22:00.003 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:59.998+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.003 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:21:59.998+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bb991d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.007 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.006+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.008 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.006+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc4fe780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.013 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.010+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.013 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.010+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc4fe100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.019 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.014+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.019 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.014+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc4fe080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.025 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.022+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.025 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.022+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc94c180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.031 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.030+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.032 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.030+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc94c380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.038 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.034+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.038 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.034+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc94c980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.044 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.042+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.044 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.042+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc94c800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.050 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.046+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.050 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.046+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc94cf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.056 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.054+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.057 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.054+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc94cd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.064 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.062+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.065 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.062+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc94cb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.072 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.066+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.072 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.066+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc94c600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.079 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.074+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.079 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.074+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc94d180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.084 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.082+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.084 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.082+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc94d380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.092 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.090+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.093 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.090+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc94d580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.099 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.094+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.099 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.094+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc94d780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.106 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.102+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.106 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.102+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc94d980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.112 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.110+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.112 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.110+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc94db80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.118 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.114+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.118 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.114+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc94dd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.126 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.122+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.126 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:00.122+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc94df80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:22:00.685 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:22:00.686 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:22:00.686 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:22:00.692 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:22:01.532 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1998324367 2023-10-20T09:22:01.536 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:22:01.538 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793721 2023-10-20T09:22:01.538 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:22:01.538 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1998324367 ]] 2023-10-20T09:22:01.538 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1998324367 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\9\9\8\3\2\4\3\6\7 ]] 2023-10-20T09:22:01.539 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:22:11.542 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:22:11.543 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:22:11.545 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:22:11.546 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:22:12.563 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1998324367 2023-10-20T09:22:12.564 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:22:12.564 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793732 2023-10-20T09:22:12.564 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:22:12.565 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1998324367 ]] 2023-10-20T09:22:12.565 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1998324367 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\9\9\8\3\2\4\3\6\7 ]] 2023-10-20T09:22:12.565 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:22:22.570 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:22:22.571 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:22:22.571 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:22:22.572 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:22:23.432 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1998324367 2023-10-20T09:22:23.433 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:22:23.433 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793743 2023-10-20T09:22:23.434 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:22:23.434 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1998324367 ]] 2023-10-20T09:22:23.434 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1998324367 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\9\9\8\3\2\4\3\6\7 ]] 2023-10-20T09:22:23.434 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:22:33.444 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:22:33.447 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:22:33.448 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:22:33.448 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:22:34.364 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/587169855 2023-10-20T09:22:34.364 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:22:34.365 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793754 2023-10-20T09:22:34.365 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:22:34.365 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/587169855 ]] 2023-10-20T09:22:34.365 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/587169855 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\9\9\8\3\2\4\3\6\7 ]] 2023-10-20T09:22:34.366 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/587169855 2023-10-20T09:22:36.068 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/587169855 until 2023-10-20T10:22:35.331841+0000 (3600 sec) 2023-10-20T09:22:36.103 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:22:36.104 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/587169855 2023-10-20T09:22:36.949 INFO:tasks.workunit.client.0.smithi150.stderr:listed 7 entries 2023-10-20T09:22:36.980 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/587169855 2023-10-20T09:22:36.980 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:22:46.981 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:22:46.981 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:22:46.982 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:22:46.984 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:22:47.942 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/587169855 2023-10-20T09:22:47.943 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:22:47.943 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793767 2023-10-20T09:22:47.944 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:22:47.944 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/587169855 ]] 2023-10-20T09:22:47.944 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/587169855 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\5\8\7\1\6\9\8\5\5 ]] 2023-10-20T09:22:47.944 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:22:57.965 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:22:57.965 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:22:57.966 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:22:57.966 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:22:58.873 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/587169855 2023-10-20T09:22:58.873 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:22:58.874 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793778 2023-10-20T09:22:58.874 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:22:58.874 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/587169855 ]] 2023-10-20T09:22:58.875 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/587169855 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\5\8\7\1\6\9\8\5\5 ]] 2023-10-20T09:22:58.875 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:23:00.002 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:59.998+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.003 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:22:59.998+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc6b1f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.006 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.002+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.006 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.002+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc688e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.008 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.006+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.008 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.006+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc95a600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.020 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.018+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.020 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.018+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc688800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.030 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.026+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.030 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.026+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc95a480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.037 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.034+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.037 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.034+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc95a180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.045 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.042+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.045 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.042+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc95ad00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.059 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.054+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.059 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.054+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc95ab80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.062 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.058+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.063 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.058+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc95b280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.068 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.066+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.068 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.066+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc95b100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.074 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.070+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.074 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.070+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc95af00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.080 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.078+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.080 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.078+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc95a980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.086 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.082+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.086 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.082+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc95b500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.092 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.090+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.093 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.090+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc95b700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.098 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.094+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.098 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.094+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc95b900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.104 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.102+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.104 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.102+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc95bb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.109 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.106+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.109 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.106+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc95bd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.114 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.110+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.114 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.110+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc95bf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.118 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.114+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.118 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.114+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc2f6100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.121 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.118+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:00.122 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:00.118+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc2f6300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:23:08.876 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:23:08.876 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:23:08.878 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:23:08.878 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:23:09.730 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/587169855 2023-10-20T09:23:09.731 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:23:09.731 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793789 2023-10-20T09:23:09.731 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:23:09.732 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/587169855 ]] 2023-10-20T09:23:09.732 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/587169855 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\5\8\7\1\6\9\8\5\5 ]] 2023-10-20T09:23:09.732 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:23:19.736 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:23:19.737 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:23:19.740 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:23:19.753 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:23:20.568 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/587169855 2023-10-20T09:23:20.568 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:23:20.569 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793800 2023-10-20T09:23:20.569 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:23:20.569 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/587169855 ]] 2023-10-20T09:23:20.570 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/587169855 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\5\8\7\1\6\9\8\5\5 ]] 2023-10-20T09:23:20.570 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:23:30.571 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:23:30.571 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:23:30.577 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:23:30.602 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:23:31.384 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/587169855 2023-10-20T09:23:31.384 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:23:31.385 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793811 2023-10-20T09:23:31.385 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:23:31.386 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/587169855 ]] 2023-10-20T09:23:31.386 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/587169855 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\5\8\7\1\6\9\8\5\5 ]] 2023-10-20T09:23:31.386 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:23:41.392 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:23:41.393 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:23:41.393 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:23:41.402 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:23:42.339 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/22457227 2023-10-20T09:23:42.340 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:23:42.341 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793822 2023-10-20T09:23:42.341 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:23:42.342 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/22457227 ]] 2023-10-20T09:23:42.342 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/22457227 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\5\8\7\1\6\9\8\5\5 ]] 2023-10-20T09:23:42.342 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/22457227 2023-10-20T09:23:44.914 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/22457227 until 2023-10-20T10:23:44.116269+0000 (3600 sec) 2023-10-20T09:23:44.958 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:23:44.961 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/22457227 2023-10-20T09:23:45.668 INFO:tasks.workunit.client.0.smithi150.stderr:listed 8 entries 2023-10-20T09:23:45.703 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/22457227 2023-10-20T09:23:45.704 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:23:55.708 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:23:55.708 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:23:55.716 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:23:55.723 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:23:56.604 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/22457227 2023-10-20T09:23:56.605 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:23:56.606 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793836 2023-10-20T09:23:56.606 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:23:56.606 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/22457227 ]] 2023-10-20T09:23:56.607 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/22457227 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\2\4\5\7\2\2\7 ]] 2023-10-20T09:23:56.607 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:24:00.002 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:59.997+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.002 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:23:59.997+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc072700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.005 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.001+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.005 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.001+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc4ffe00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.009 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.005+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.009 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.005+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc6b1100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.016 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.013+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.016 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.013+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc95bf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.021 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.017+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.021 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.017+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bb7c7e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.027 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.025+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.027 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.025+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc920680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.032 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.029+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.032 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.029+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bb8f0b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.035 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.033+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.035 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.033+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bbf9c980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.040 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.037+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.040 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.037+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc2f7900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.046 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.041+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.046 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.041+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc2f7a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.054 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.049+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.054 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.049+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc2f6680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.062 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.057+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.062 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.057+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bb8f0000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.068 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.065+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.069 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.065+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc005280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.077 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.073+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.077 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.073+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc2f6180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.086 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.081+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.087 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.081+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bca78180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.091 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.089+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.092 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.089+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bca78380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.099 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.097+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.100 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.097+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bca78580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.106 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.101+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.106 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.101+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bca78780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.112 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.109+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.113 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.109+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bca78980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.120 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.117+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:00.120 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:24:00.117+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bca78b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:24:06.607 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:24:06.608 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:24:06.614 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:24:06.628 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:24:07.651 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/22457227 2023-10-20T09:24:07.651 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:24:07.653 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793847 2023-10-20T09:24:07.653 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:24:07.653 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/22457227 ]] 2023-10-20T09:24:07.653 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/22457227 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\2\4\5\7\2\2\7 ]] 2023-10-20T09:24:07.653 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:24:14.186 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:24:14.185+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:24:14.186 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:24:14.185+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T09:24:14.186 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:24:14.185+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:24:14.187 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:24:14.185+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab4f70 2023-10-20T09:24:14.187 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:24:14.185+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T09:24:14.187 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:24:14.185+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:24:14.187 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:24:14.185+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:24:14.187 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:24:14.185+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T09:24:14.188 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:24:14.185+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:24:14.188 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:24:14.185+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:24:14.188 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:24:14.185+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2750 2023-10-20T09:24:14.188 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:24:14.185+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2820 2023-10-20T09:24:14.188 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:24:14.185+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:24:14.189 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:24:14.185+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:24:17.654 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:24:17.663 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:24:17.663 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:24:17.691 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:24:18.652 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/22457227 2023-10-20T09:24:18.653 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:24:18.653 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793858 2023-10-20T09:24:18.653 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:24:18.654 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/22457227 ]] 2023-10-20T09:24:18.654 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/22457227 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\2\4\5\7\2\2\7 ]] 2023-10-20T09:24:18.654 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:24:28.659 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:24:28.660 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:24:28.660 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:24:28.665 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:24:29.579 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/22457227 2023-10-20T09:24:29.579 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:24:29.580 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793869 2023-10-20T09:24:29.580 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:24:29.580 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/22457227 ]] 2023-10-20T09:24:29.581 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/22457227 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\2\4\5\7\2\2\7 ]] 2023-10-20T09:24:29.581 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:24:39.582 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:24:39.582 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:24:39.590 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:24:39.598 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:24:40.458 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/22457227 2023-10-20T09:24:40.458 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:24:40.459 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793880 2023-10-20T09:24:40.459 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:24:40.459 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/22457227 ]] 2023-10-20T09:24:40.459 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/22457227 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\2\4\5\7\2\2\7 ]] 2023-10-20T09:24:40.460 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:24:50.461 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:24:50.495 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:24:50.496 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:24:50.496 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:24:51.463 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/22457227 2023-10-20T09:24:51.463 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:24:51.464 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793891 2023-10-20T09:24:51.464 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:24:51.464 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/22457227 ]] 2023-10-20T09:24:51.464 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/22457227 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\2\4\5\7\2\2\7 ]] 2023-10-20T09:24:51.464 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:25:01.478 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:25:01.478 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:25:01.480 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:25:01.491 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:25:01.725 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:25:01.724+0000 7f3810eae640 -1 reset not still connected to 0x5632dbbda270 2023-10-20T09:25:02.089 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:25:02.088+0000 7f3810eae640 -1 reset not still connected to 0x5632dbbda820 2023-10-20T09:25:02.090 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:25:02.088+0000 7f3810eae640 -1 reset not still connected to 0x5632dbbda8f0 2023-10-20T09:25:02.406 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2268406928 2023-10-20T09:25:02.406 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:25:02.407 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793902 2023-10-20T09:25:02.407 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:25:02.407 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2268406928 ]] 2023-10-20T09:25:02.408 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2268406928 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\2\4\5\7\2\2\7 ]] 2023-10-20T09:25:02.408 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/2268406928 2023-10-20T09:25:02.928 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:02.928+0000 7f28a3c2f640 -1 librbd::ImageWatcher: 0x5610bc4ea300 image watch failed: 94629886949376, (107) Transport endpoint is not connected 2023-10-20T09:25:02.928 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:02.928+0000 7f28a3c2f640 -1 librbd::Watcher: 0x5610bc4ea300 handle_error: handle=94629886949376: (107) Transport endpoint is not connected 2023-10-20T09:25:02.929 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:02.928+0000 7f28a342e640 -1 librbd::watcher::RewatchRequest: 0x5610bc27b3b0 handle_unwatch client blocklisted 2023-10-20T09:25:02.929 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:02.928+0000 7f28a342e640 -1 librbd::Watcher: 0x5610bc4ea300 handle_rewatch: client blocklisted 2023-10-20T09:25:02.929 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:02.928+0000 7f28a342e640 -1 librbd::ImageWatcher: 0x5610bbf2d500 image watch failed: 94629760116736, (107) Transport endpoint is not connected 2023-10-20T09:25:02.929 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:02.928+0000 7f28a342e640 -1 librbd::Watcher: 0x5610bbf2d500 handle_error: handle=94629760116736: (107) Transport endpoint is not connected 2023-10-20T09:25:02.930 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:02.928+0000 7f28a342e640 -1 librbd::watcher::RewatchRequest: 0x5610bc27b400 handle_unwatch client blocklisted 2023-10-20T09:25:02.930 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:02.928+0000 7f28a342e640 -1 librbd::Watcher: 0x5610bbf2d500 handle_rewatch: client blocklisted 2023-10-20T09:25:03.024 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.024+0000 7f28a342e640 -1 librbd::ImageWatcher: 0x5610bc933800 image watch failed: 94629886943232, (107) Transport endpoint is not connected 2023-10-20T09:25:03.025 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.024+0000 7f28a342e640 -1 librbd::Watcher: 0x5610bc933800 handle_error: handle=94629886943232: (107) Transport endpoint is not connected 2023-10-20T09:25:03.025 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.024+0000 7f28a342e640 -1 librbd::ImageWatcher: 0x5610bbf2d200 image watch failed: 94629876402176, (107) Transport endpoint is not connected 2023-10-20T09:25:03.025 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.024+0000 7f28a342e640 -1 librbd::Watcher: 0x5610bbf2d200 handle_error: handle=94629876402176: (107) Transport endpoint is not connected 2023-10-20T09:25:03.415 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.416+0000 7f28a342e640 -1 librbd::watcher::RewatchRequest: 0x5610bb98f130 handle_unwatch client blocklisted 2023-10-20T09:25:03.416 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.416+0000 7f28a342e640 -1 librbd::Watcher: 0x5610bc933800 handle_rewatch: client blocklisted 2023-10-20T09:25:03.416 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.416+0000 7f28a342e640 -1 librbd::ManagedLock: 0x5610bb7ee1b8 send_acquire_lock: watcher not registered - client blocklisted 2023-10-20T09:25:03.416 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.416+0000 7f28a342e640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5610bcd0a000 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:25:03.416 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.416+0000 7f28a342e640 -1 librbd::io::AioCompletion: 0x5610bcd02c60 fail: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:25:03.416 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.416+0000 7f28a342e640 -1 librbd::watcher::RewatchRequest: 0x5610bb98e370 handle_unwatch client blocklisted 2023-10-20T09:25:03.417 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.416+0000 7f28a342e640 -1 librbd::Watcher: 0x5610bbf2d200 handle_rewatch: client blocklisted 2023-10-20T09:25:03.922 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/2268406928 until 2023-10-20T10:25:03.120120+0000 (3600 sec) 2023-10-20T09:25:03.929 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.928+0000 7f28a342e640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:25:03.930 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.928+0000 7f28a342e640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:25:03.930 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.928+0000 7f28a342e640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5610bcd0a340 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:25:03.930 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.928+0000 7f28a342e640 -1 librbd::io::AioCompletion: 0x5610bcacc840 fail: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:25:03.930 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.928+0000 7f28a342e640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5610bcd0a0d0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:25:03.931 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.928+0000 7f28a342e640 -1 librbd::io::AioCompletion: 0x5610bcc271e0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:25:03.931 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.932+0000 7f28a342e640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:25:03.932 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.932+0000 7f28a342e640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:25:03.932 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.932+0000 7f28a342e640 -1 librbd::io::AioCompletion: 0x5610bca42000 fail: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:25:03.932 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.932+0000 7f28a342e640 -1 librbd::io::AioCompletion: 0x5610bcd02c60 fail: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:25:03.932 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.932+0000 7f28a3c2f640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5610bbbb7d00 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:25:03.933 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.932+0000 7f28a342e640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5610bcbedac0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:25:03.933 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.932+0000 7f28a3c2f640 -1 librbd::managed_lock::ReleaseRequest: 0x5610bc792e40 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:25:03.933 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.932+0000 7f28a342e640 -1 librbd::managed_lock::ReleaseRequest: 0x5610bc200180 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:25:03.945 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.944+0000 7f28a3c2f640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:25:03.945 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.944+0000 7f28a3c2f640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5610bcd0a410 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:25:03.945 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.944+0000 7f28a3c2f640 -1 librbd::io::AioCompletion: 0x5610bcab0f20 fail: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:25:03.946 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.948+0000 7f28a3c2f640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:25:03.947 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.948+0000 7f28a3c2f640 -1 librbd::io::AioCompletion: 0x5610bcd02c60 fail: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:25:03.947 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.948+0000 7f28a3c2f640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5610bcd11e80 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:25:03.947 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:25:03.948+0000 7f28a3c2f640 -1 librbd::managed_lock::ReleaseRequest: 0x5610bc660240 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:25:03.953 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:25:03.953 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/2268406928 2023-10-20T09:25:04.576 INFO:tasks.workunit.client.0.smithi150.stderr:listed 9 entries 2023-10-20T09:25:04.605 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/2268406928 2023-10-20T09:25:04.605 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:25:14.607 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:25:14.610 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:25:14.614 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:25:14.624 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:25:15.527 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2268406928 2023-10-20T09:25:15.527 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:25:15.528 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793915 2023-10-20T09:25:15.528 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:25:15.528 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2268406928 ]] 2023-10-20T09:25:15.529 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2268406928 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\2\6\8\4\0\6\9\2\8 ]] 2023-10-20T09:25:15.529 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:25:25.529 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:25:25.530 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:25:25.530 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:25:25.531 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:25:26.390 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2268406928 2023-10-20T09:25:26.390 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:25:26.391 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793926 2023-10-20T09:25:26.391 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:25:26.391 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2268406928 ]] 2023-10-20T09:25:26.391 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2268406928 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\2\6\8\4\0\6\9\2\8 ]] 2023-10-20T09:25:26.391 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:25:36.392 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:25:36.406 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:25:36.409 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:25:36.414 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:25:37.378 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2268406928 2023-10-20T09:25:37.379 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:25:37.380 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793937 2023-10-20T09:25:37.380 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:25:37.380 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2268406928 ]] 2023-10-20T09:25:37.381 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2268406928 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\2\6\8\4\0\6\9\2\8 ]] 2023-10-20T09:25:37.381 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:25:47.382 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:25:47.382 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:25:47.390 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:25:47.402 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:25:48.337 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2268406928 2023-10-20T09:25:48.338 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:25:48.338 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793948 2023-10-20T09:25:48.338 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:25:48.339 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2268406928 ]] 2023-10-20T09:25:48.339 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2268406928 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\2\6\8\4\0\6\9\2\8 ]] 2023-10-20T09:25:48.339 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:25:58.340 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:25:58.340 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:25:58.346 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:25:58.347 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:25:59.206 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2268406928 2023-10-20T09:25:59.216 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:25:59.217 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793959 2023-10-20T09:25:59.217 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:25:59.217 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2268406928 ]] 2023-10-20T09:25:59.217 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2268406928 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\2\6\8\4\0\6\9\2\8 ]] 2023-10-20T09:25:59.218 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:26:09.219 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:26:09.219 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:26:09.223 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:26:09.239 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:26:10.098 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1670666381 2023-10-20T09:26:10.099 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:26:10.099 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793970 2023-10-20T09:26:10.100 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:26:10.100 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1670666381 ]] 2023-10-20T09:26:10.100 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1670666381 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\2\6\8\4\0\6\9\2\8 ]] 2023-10-20T09:26:10.100 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/1670666381 2023-10-20T09:26:11.931 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/1670666381 until 2023-10-20T10:26:11.139078+0000 (3600 sec) 2023-10-20T09:26:11.975 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/1670666381 2023-10-20T09:26:11.976 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:26:12.755 INFO:tasks.workunit.client.0.smithi150.stderr:listed 10 entries 2023-10-20T09:26:12.794 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/1670666381 2023-10-20T09:26:12.795 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:26:22.797 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:26:22.797 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:26:22.797 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:26:22.797 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:26:23.823 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1670666381 2023-10-20T09:26:23.833 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:26:23.834 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793983 2023-10-20T09:26:23.834 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:26:23.834 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1670666381 ]] 2023-10-20T09:26:23.835 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1670666381 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\6\7\0\6\6\6\3\8\1 ]] 2023-10-20T09:26:23.835 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:26:33.838 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:26:33.841 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:26:33.842 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:26:33.850 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:26:34.682 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1670666381 2023-10-20T09:26:34.683 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:26:34.683 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697793994 2023-10-20T09:26:34.684 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:26:34.684 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1670666381 ]] 2023-10-20T09:26:34.684 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1670666381 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\6\7\0\6\6\6\3\8\1 ]] 2023-10-20T09:26:34.684 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:26:44.685 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:26:44.686 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:26:44.694 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:26:44.705 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:26:45.639 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1670666381 2023-10-20T09:26:45.642 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:26:45.644 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794005 2023-10-20T09:26:45.644 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:26:45.644 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1670666381 ]] 2023-10-20T09:26:45.644 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1670666381 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\6\7\0\6\6\6\3\8\1 ]] 2023-10-20T09:26:45.644 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:26:55.660 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:26:55.661 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:26:55.663 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:26:55.665 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:26:56.569 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1670666381 2023-10-20T09:26:56.575 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:26:56.582 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794016 2023-10-20T09:26:56.582 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:26:56.583 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1670666381 ]] 2023-10-20T09:26:56.583 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1670666381 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\6\7\0\6\6\6\3\8\1 ]] 2023-10-20T09:26:56.583 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:27:00.001 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:26:59.998+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.001 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:26:59.998+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc4fe300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.011 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.010+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.012 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.010+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc2f6d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.018 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.018+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.018 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.018+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc2f7e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.024 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.022+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.024 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.022+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc2f7c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.042 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.042+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.043 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.042+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc2f7b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.048 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.046+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.048 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.046+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bb8f0100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.060 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.058+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.060 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.058+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc2f7800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.065 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.062+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.065 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.062+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc72a180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.070 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.070+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.070 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.070+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bccc2580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.074 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.074+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.075 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.074+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bccc2980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.085 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.082+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.086 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.082+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc073880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.087 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.086+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.087 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.086+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc073700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.089 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.086+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.089 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.086+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bccc3e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.098 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.098+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.098 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.098+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcada000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.102 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.102+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.102 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.102+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcada200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.111 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.110+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.111 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.110+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcada400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.120 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.118+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.121 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.118+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcada600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.129 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.126+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.129 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.126+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bccc2980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.133 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.130+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.133 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.130+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bccc3e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.148 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.146+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:00.148 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:27:00.146+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc072c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:27:06.594 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:27:06.598 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:27:06.601 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:27:06.608 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:27:07.485 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1670666381 2023-10-20T09:27:07.486 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:27:07.486 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794027 2023-10-20T09:27:07.487 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:27:07.487 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1670666381 ]] 2023-10-20T09:27:07.487 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1670666381 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\6\7\0\6\6\6\3\8\1 ]] 2023-10-20T09:27:07.487 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:27:11.658 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:11.654+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:27:11.658 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:11.654+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:27:17.488 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:27:17.489 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:27:17.489 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:27:17.496 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:27:18.435 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/569130003 2023-10-20T09:27:18.438 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:27:18.439 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794038 2023-10-20T09:27:18.439 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:27:18.439 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/569130003 ]] 2023-10-20T09:27:18.440 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/569130003 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\6\7\0\6\6\6\3\8\1 ]] 2023-10-20T09:27:18.440 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/569130003 2023-10-20T09:27:20.976 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/569130003 until 2023-10-20T10:27:20.183603+0000 (3600 sec) 2023-10-20T09:27:21.023 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/569130003 2023-10-20T09:27:21.024 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:27:21.776 INFO:tasks.workunit.client.0.smithi150.stderr:listed 11 entries 2023-10-20T09:27:21.807 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/569130003 2023-10-20T09:27:21.808 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:27:31.812 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:27:31.818 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:27:31.819 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:27:31.819 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:27:31.838 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:31.834+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:27:32.678 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/569130003 2023-10-20T09:27:32.678 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:27:32.679 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794052 2023-10-20T09:27:32.679 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:27:32.679 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/569130003 ]] 2023-10-20T09:27:32.679 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/569130003 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\5\6\9\1\3\0\0\0\3 ]] 2023-10-20T09:27:32.680 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:27:40.109 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:40.106+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:27:40.109 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:40.106+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T09:27:40.110 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:40.106+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:27:40.110 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:40.106+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab4f70 2023-10-20T09:27:40.110 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:40.106+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T09:27:40.111 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:40.106+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:27:40.111 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:40.106+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:27:40.111 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:40.106+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T09:27:40.112 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:40.106+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:27:40.112 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:40.106+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:27:40.112 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:40.106+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2750 2023-10-20T09:27:42.683 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:27:42.685 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:27:42.690 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:27:42.693 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:27:43.583 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/569130003 2023-10-20T09:27:43.589 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:27:43.591 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794063 2023-10-20T09:27:43.591 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:27:43.591 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/569130003 ]] 2023-10-20T09:27:43.591 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/569130003 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\5\6\9\1\3\0\0\0\3 ]] 2023-10-20T09:27:43.591 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:27:44.879 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:44.878+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:27:44.882 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:44.882+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T09:27:44.882 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:44.882+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:27:44.900 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:44.894+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab4f70 2023-10-20T09:27:44.900 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:44.894+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T09:27:44.900 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:44.894+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:27:44.900 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:44.894+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:27:44.901 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:44.894+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T09:27:44.901 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:44.894+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:27:44.901 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:44.894+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:27:44.901 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:44.894+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2750 2023-10-20T09:27:44.901 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:27:44.894+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2820 2023-10-20T09:27:53.601 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:27:53.602 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:27:53.602 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:27:53.603 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:27:54.510 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/569130003 2023-10-20T09:27:54.511 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:27:54.512 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794074 2023-10-20T09:27:54.512 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:27:54.512 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/569130003 ]] 2023-10-20T09:27:54.512 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/569130003 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\5\6\9\1\3\0\0\0\3 ]] 2023-10-20T09:27:54.512 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:28:00.003 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.002+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.003 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.002+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc072300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.006 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.006+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.006 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.006+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc921100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.017 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.018+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.018 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.018+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.018 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.018+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc6b1700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.018 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.018+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc921c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.020 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.018+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.021 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.018+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bccc3a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.023 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.022+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.023 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.022+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bccc2900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.027 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.026+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.027 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.026+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bccc3f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.032 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.030+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.033 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.030+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bccc3500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.038 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.038+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.038 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.038+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcada680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.044 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.042+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.044 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.042+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcada400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.049 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.050+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.049 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.050+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcada200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.056 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.054+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.056 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.054+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bccc2d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.059 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.058+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.059 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.058+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcada800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.061 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.062+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.062 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.062+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcadaa00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.064 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.062+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.064 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.062+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcadac00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.071 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.070+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.071 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.070+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcadae00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.085 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.082+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.085 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.082+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcadb000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.091 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.090+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.092 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.090+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcadb200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.098 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.098+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.098 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.098+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcadbc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.104 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.102+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:00.105 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:28:00.102+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcadba80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:28:04.516 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:28:04.516 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:28:04.517 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:28:04.520 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:28:05.347 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/569130003 2023-10-20T09:28:05.347 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:28:05.348 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794085 2023-10-20T09:28:05.348 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:28:05.349 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/569130003 ]] 2023-10-20T09:28:05.349 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/569130003 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\5\6\9\1\3\0\0\0\3 ]] 2023-10-20T09:28:05.349 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:28:15.354 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:28:15.359 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:28:15.362 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:28:15.365 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:28:16.389 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/569130003 2023-10-20T09:28:16.389 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:28:16.390 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794096 2023-10-20T09:28:16.390 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:28:16.390 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/569130003 ]] 2023-10-20T09:28:16.391 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/569130003 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\5\6\9\1\3\0\0\0\3 ]] 2023-10-20T09:28:16.391 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:28:26.395 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:28:26.396 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:28:26.396 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:28:26.407 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:28:27.228 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2805376037 2023-10-20T09:28:27.228 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:28:27.229 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794107 2023-10-20T09:28:27.229 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:28:27.229 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2805376037 ]] 2023-10-20T09:28:27.229 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2805376037 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\5\6\9\1\3\0\0\0\3 ]] 2023-10-20T09:28:27.230 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/2805376037 2023-10-20T09:28:28.996 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/2805376037 until 2023-10-20T10:28:28.276815+0000 (3600 sec) 2023-10-20T09:28:29.035 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:28:29.035 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/2805376037 2023-10-20T09:28:29.801 INFO:tasks.workunit.client.0.smithi150.stderr:listed 12 entries 2023-10-20T09:28:29.842 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/2805376037 2023-10-20T09:28:29.843 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:28:39.847 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:28:39.848 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:28:39.856 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:28:39.866 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:28:40.818 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2805376037 2023-10-20T09:28:40.818 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:28:40.819 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794120 2023-10-20T09:28:40.819 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:28:40.819 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2805376037 ]] 2023-10-20T09:28:40.820 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2805376037 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\8\0\5\3\7\6\0\3\7 ]] 2023-10-20T09:28:40.820 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:28:50.832 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:28:50.833 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:28:50.833 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:28:50.838 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:28:51.681 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2805376037 2023-10-20T09:28:51.681 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:28:51.682 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794131 2023-10-20T09:28:51.683 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:28:51.683 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2805376037 ]] 2023-10-20T09:28:51.683 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2805376037 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\8\0\5\3\7\6\0\3\7 ]] 2023-10-20T09:28:51.684 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:29:00.004 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.001+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.004 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.001+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.005 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.001+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc411280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.005 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.001+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc411080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.005 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.005+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.006 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.005+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc411480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.008 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.005+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.009 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.005+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc411680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.019 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.017+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.019 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.017+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc411880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.023 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.021+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.023 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.021+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc411a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.032 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.029+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.032 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.029+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc411c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.038 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.037+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.038 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.037+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc411e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.043 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.041+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.043 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.041+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcb32080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.050 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.049+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.050 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.049+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcb32280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.056 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.053+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.056 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.053+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcb32480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.062 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.061+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.062 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.061+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcb32680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.068 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.065+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.068 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.065+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcb32e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.074 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.073+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.074 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.073+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcb32c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.080 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.077+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.080 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.077+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcb32a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.086 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.085+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.087 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.085+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcb32900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.092 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.089+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.092 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.089+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcb33080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.099 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.097+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.102 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.097+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcb33280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.106 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.105+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.106 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.105+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcb33480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.113 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.113+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:00.113 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:29:00.113+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcb33680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:29:01.684 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:29:01.685 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:29:01.685 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:29:01.688 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:29:02.670 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2805376037 2023-10-20T09:29:02.670 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:29:02.671 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794142 2023-10-20T09:29:02.671 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:29:02.671 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2805376037 ]] 2023-10-20T09:29:02.671 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2805376037 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\8\0\5\3\7\6\0\3\7 ]] 2023-10-20T09:29:02.671 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:29:12.677 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:29:12.678 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:29:12.678 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:29:12.701 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:29:13.698 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2805376037 2023-10-20T09:29:13.699 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:29:13.700 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794153 2023-10-20T09:29:13.700 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:29:13.700 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2805376037 ]] 2023-10-20T09:29:13.700 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2805376037 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\8\0\5\3\7\6\0\3\7 ]] 2023-10-20T09:29:13.701 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:29:23.702 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:29:23.702 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:29:23.704 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:29:23.716 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:29:24.624 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2805376037 2023-10-20T09:29:24.624 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:29:24.625 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794164 2023-10-20T09:29:24.625 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:29:24.625 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2805376037 ]] 2023-10-20T09:29:24.625 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2805376037 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\8\0\5\3\7\6\0\3\7 ]] 2023-10-20T09:29:24.626 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:29:34.629 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:29:34.629 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:29:34.634 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:29:34.640 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:29:35.625 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/260822388 2023-10-20T09:29:35.626 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:29:35.627 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794175 2023-10-20T09:29:35.627 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:29:35.627 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/260822388 ]] 2023-10-20T09:29:35.627 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/260822388 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\8\0\5\3\7\6\0\3\7 ]] 2023-10-20T09:29:35.628 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/260822388 2023-10-20T09:29:38.026 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/260822388 until 2023-10-20T10:29:37.305164+0000 (3600 sec) 2023-10-20T09:29:38.054 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:29:38.059 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/260822388 2023-10-20T09:29:38.795 INFO:tasks.workunit.client.0.smithi150.stderr:listed 13 entries 2023-10-20T09:29:38.825 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/260822388 2023-10-20T09:29:38.825 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:29:41.216 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:29:41.209+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:29:41.217 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:29:41.217+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:29:48.827 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:29:48.827 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:29:48.832 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:29:48.842 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:29:49.717 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/260822388 2023-10-20T09:29:49.718 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:29:49.718 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794189 2023-10-20T09:29:49.718 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:29:49.719 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/260822388 ]] 2023-10-20T09:29:49.719 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/260822388 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\6\0\8\2\2\3\8\8 ]] 2023-10-20T09:29:49.719 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:29:57.261 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:29:57.260+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:29:57.262 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:29:57.260+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:29:59.732 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:29:59.732 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:29:59.733 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:29:59.740 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:30:00.001 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.004+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.002 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.004+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcadb780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.005 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.008+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.006 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.008+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcadbe00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.013 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.016+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.013 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.016+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcadb480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.021 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.020+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.021 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.020+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcadac80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.031 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.032+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.032 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.032+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcb33b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.038 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.040+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.038 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.040+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcb33880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.045 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.048+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.045 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.048+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcb33e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.073 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.076+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.074 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.076+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcc46000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.085 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.088+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.085 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.088+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcc46200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.093 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.096+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.093 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.096+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcc46400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.096 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.100+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.096 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.100+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcc46600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.112 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.112+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.112 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.112+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcc46800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.118 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.120+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.118 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.120+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcc46a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.125 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.128+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.125 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.128+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcc46c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.132 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.136+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.132 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.136+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcc47400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.139 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.140+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.139 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.140+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcc47280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.149 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.152+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.149 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.152+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcc47080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.155 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.156+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.155 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.156+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcc46a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.160 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.164+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.161 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.164+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcc46780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.179 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.180+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.179 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:30:00.180+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcc46580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:00.357 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:30:00.360+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:30:00.621 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/260822388 2023-10-20T09:30:00.622 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:30:00.622 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794200 2023-10-20T09:30:00.623 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:30:00.623 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/260822388 ]] 2023-10-20T09:30:00.623 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/260822388 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\6\0\8\2\2\3\8\8 ]] 2023-10-20T09:30:00.623 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:30:10.629 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:30:10.630 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:30:10.630 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:30:10.648 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:30:11.565 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/260822388 2023-10-20T09:30:11.566 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:30:11.567 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794211 2023-10-20T09:30:11.567 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:30:11.567 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/260822388 ]] 2023-10-20T09:30:11.568 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/260822388 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\6\0\8\2\2\3\8\8 ]] 2023-10-20T09:30:11.568 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:30:21.573 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:30:21.577 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:30:21.580 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:30:21.584 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:30:22.566 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/260822388 2023-10-20T09:30:22.566 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:30:22.567 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794222 2023-10-20T09:30:22.568 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:30:22.568 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/260822388 ]] 2023-10-20T09:30:22.568 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/260822388 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\6\0\8\2\2\3\8\8 ]] 2023-10-20T09:30:22.568 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:30:30.074 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:30:30.076+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:30:30.089 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:30:30.088+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:30:32.568 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:30:32.569 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:30:32.572 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:30:32.609 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:30:33.554 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/260822388 2023-10-20T09:30:33.556 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:30:33.557 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794233 2023-10-20T09:30:33.557 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:30:33.558 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/260822388 ]] 2023-10-20T09:30:33.558 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/260822388 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\6\0\8\2\2\3\8\8 ]] 2023-10-20T09:30:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:30:43.559 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:30:43.559 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:30:43.570 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:30:43.578 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:30:44.399 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/260822388 2023-10-20T09:30:44.400 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:30:44.482 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794244 2023-10-20T09:30:44.483 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:30:44.483 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/260822388 ]] 2023-10-20T09:30:44.483 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/260822388 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\6\0\8\2\2\3\8\8 ]] 2023-10-20T09:30:44.483 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:30:54.491 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:30:54.492 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:30:54.499 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:30:54.500 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:30:55.667 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1360796600 2023-10-20T09:30:55.668 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:30:55.672 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794255 2023-10-20T09:30:55.672 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:30:55.672 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1360796600 ]] 2023-10-20T09:30:55.673 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1360796600 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\6\0\8\2\2\3\8\8 ]] 2023-10-20T09:30:55.673 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/1360796600 2023-10-20T09:30:58.019 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/1360796600 until 2023-10-20T10:30:57.283855+0000 (3600 sec) 2023-10-20T09:30:58.072 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:30:58.072 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/1360796600 2023-10-20T09:30:58.841 INFO:tasks.workunit.client.0.smithi150.stderr:listed 14 entries 2023-10-20T09:30:58.868 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/1360796600 2023-10-20T09:30:58.868 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:30:59.999 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.000+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:30:59.999 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.000+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bbf9d300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.003 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.004+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.003 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.004+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc72ad80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.008 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.012+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.008 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.012+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc6b0d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.014 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.016+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.015 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.016+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc72ae00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.019 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.020+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.019 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.020+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc210700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.024 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.028+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.025 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.028+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc005380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.042 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.044+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.042 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.044+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc920a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.057 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.060+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.057 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.060+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc4feb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.073 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.076+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.074 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.076+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bb990800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.078 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.080+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.079 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.080+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc005000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.085 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.088+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.085 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.088+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc921980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.093 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.092+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.093 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.092+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc921600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.093 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.096+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.093 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.096+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcada980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.098 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.100+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.098 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.100+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcadb480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.103 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.108+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.104 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.108+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bb990700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.110 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.112+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.110 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.112+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc2f7c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.116 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.120+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.116 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.120+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc2f6100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.123 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.124+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.123 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.124+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc2f7d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.129 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.132+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.130 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.132+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcb32b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.137 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.140+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:00.137 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:31:00.140+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcb32500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:31:08.892 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:31:08.892 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:31:08.898 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:31:08.898 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:31:09.735 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1360796600 2023-10-20T09:31:09.735 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:31:09.735 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794269 2023-10-20T09:31:09.736 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:31:09.736 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1360796600 ]] 2023-10-20T09:31:09.736 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1360796600 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\3\6\0\7\9\6\6\0\0 ]] 2023-10-20T09:31:09.736 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:31:19.747 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:31:19.747 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:31:19.747 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:31:19.748 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:31:21.273 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1360796600 2023-10-20T09:31:21.281 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:31:21.284 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794281 2023-10-20T09:31:21.284 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:31:21.284 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1360796600 ]] 2023-10-20T09:31:21.284 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1360796600 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\3\6\0\7\9\6\6\0\0 ]] 2023-10-20T09:31:21.285 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:31:31.296 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:31:31.299 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:31:31.312 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:31:31.318 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:31:32.284 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1360796600 2023-10-20T09:31:32.284 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:31:32.285 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794292 2023-10-20T09:31:32.286 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:31:32.286 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1360796600 ]] 2023-10-20T09:31:32.286 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1360796600 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\3\6\0\7\9\6\6\0\0 ]] 2023-10-20T09:31:32.286 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:31:42.292 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:31:42.292 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:31:42.293 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:31:42.294 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:31:43.272 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1360796600 2023-10-20T09:31:43.285 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:31:43.290 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794303 2023-10-20T09:31:43.291 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:31:43.291 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1360796600 ]] 2023-10-20T09:31:43.291 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1360796600 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\3\6\0\7\9\6\6\0\0 ]] 2023-10-20T09:31:43.291 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:31:53.298 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:31:53.298 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:31:53.302 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:31:53.303 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:31:54.247 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1360796600 2023-10-20T09:31:54.247 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:31:54.248 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794314 2023-10-20T09:31:54.248 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:31:54.248 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1360796600 ]] 2023-10-20T09:31:54.249 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1360796600 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\3\6\0\7\9\6\6\0\0 ]] 2023-10-20T09:31:54.249 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:32:02.097 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:02.099+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T09:32:02.097 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:02.099+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:32:02.098 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:02.099+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T09:32:02.098 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:02.099+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:32:02.099 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:02.099+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:32:02.113 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:02.115+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T09:32:02.113 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:02.115+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:32:02.113 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:02.115+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:32:02.114 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:02.115+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2750 2023-10-20T09:32:04.251 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:32:04.251 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:32:04.252 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:32:04.267 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:32:05.075 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/840525808 2023-10-20T09:32:05.076 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:32:05.076 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794325 2023-10-20T09:32:05.077 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:32:05.077 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/840525808 ]] 2023-10-20T09:32:05.078 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/840525808 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\3\6\0\7\9\6\6\0\0 ]] 2023-10-20T09:32:05.079 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/840525808 2023-10-20T09:32:07.645 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/840525808 until 2023-10-20T10:32:06.810373+0000 (3600 sec) 2023-10-20T09:32:07.674 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:32:07.675 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/840525808 2023-10-20T09:32:08.415 INFO:tasks.workunit.client.0.smithi150.stderr:listed 15 entries 2023-10-20T09:32:08.439 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/840525808 2023-10-20T09:32:08.439 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:32:18.441 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:32:18.443 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:32:18.446 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:32:18.451 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:32:19.319 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/840525808 2023-10-20T09:32:19.320 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:32:19.321 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794339 2023-10-20T09:32:19.321 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:32:19.322 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/840525808 ]] 2023-10-20T09:32:19.322 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/840525808 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\8\4\0\5\2\5\8\0\8 ]] 2023-10-20T09:32:19.322 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:32:28.231 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:28.230+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:32:28.238 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:28.230+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T09:32:28.238 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:28.230+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:32:28.238 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:28.230+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab4f70 2023-10-20T09:32:28.238 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:28.230+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T09:32:28.239 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:28.230+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:32:28.239 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:28.230+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:32:28.239 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:28.230+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T09:32:28.239 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:28.230+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:32:28.239 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:28.230+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:32:28.240 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:28.230+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2750 2023-10-20T09:32:28.240 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:28.230+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2820 2023-10-20T09:32:29.321 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:32:29.322 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:32:29.324 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:32:29.331 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:32:30.208 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/840525808 2023-10-20T09:32:30.208 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:32:30.208 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794350 2023-10-20T09:32:30.209 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:32:30.209 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/840525808 ]] 2023-10-20T09:32:30.209 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/840525808 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\8\4\0\5\2\5\8\0\8 ]] 2023-10-20T09:32:30.209 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:32:40.216 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:32:40.217 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:32:40.218 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:32:40.238 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:32:41.077 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/840525808 2023-10-20T09:32:41.077 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:32:41.078 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794361 2023-10-20T09:32:41.078 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:32:41.078 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/840525808 ]] 2023-10-20T09:32:41.079 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/840525808 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\8\4\0\5\2\5\8\0\8 ]] 2023-10-20T09:32:41.079 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:32:50.752 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:50.750+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:32:51.079 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:32:51.083 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:32:51.089 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:32:51.089 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:32:51.945 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/840525808 2023-10-20T09:32:51.946 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:32:51.946 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794371 2023-10-20T09:32:51.947 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:32:51.947 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/840525808 ]] 2023-10-20T09:32:51.948 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/840525808 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\8\4\0\5\2\5\8\0\8 ]] 2023-10-20T09:32:51.948 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:32:53.403 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:53.402+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:32:53.404 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:53.402+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T09:32:53.404 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:53.402+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:32:53.405 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:53.402+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab4f70 2023-10-20T09:32:53.405 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:53.402+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T09:32:53.405 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:53.402+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:32:53.405 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:53.402+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:32:53.405 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:53.402+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T09:32:53.406 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:53.402+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:32:53.406 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:53.402+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:32:53.406 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:53.402+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2750 2023-10-20T09:32:53.406 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:32:53.402+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2820 2023-10-20T09:33:01.954 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:33:01.954 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:33:01.955 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:33:01.983 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:33:02.901 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/840525808 2023-10-20T09:33:02.902 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:33:02.902 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794382 2023-10-20T09:33:02.903 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:33:02.903 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/840525808 ]] 2023-10-20T09:33:02.903 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/840525808 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\8\4\0\5\2\5\8\0\8 ]] 2023-10-20T09:33:02.903 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:33:12.904 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:33:12.905 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:33:12.907 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:33:12.911 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:33:13.789 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2500377447 2023-10-20T09:33:13.789 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:33:13.790 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794393 2023-10-20T09:33:13.790 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:33:13.790 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2500377447 ]] 2023-10-20T09:33:13.791 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2500377447 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\8\4\0\5\2\5\8\0\8 ]] 2023-10-20T09:33:13.791 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/2500377447 2023-10-20T09:33:16.045 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/2500377447 until 2023-10-20T10:33:15.263283+0000 (3600 sec) 2023-10-20T09:33:16.099 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:33:16.113 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/2500377447 2023-10-20T09:33:16.816 INFO:tasks.workunit.client.0.smithi150.stderr:listed 16 entries 2023-10-20T09:33:16.844 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/2500377447 2023-10-20T09:33:16.844 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:33:26.846 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:33:26.849 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:33:26.857 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:33:26.864 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:33:27.866 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2500377447 2023-10-20T09:33:27.866 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:33:27.867 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794407 2023-10-20T09:33:27.867 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:33:27.867 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2500377447 ]] 2023-10-20T09:33:27.867 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2500377447 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\5\0\0\3\7\7\4\4\7 ]] 2023-10-20T09:33:27.867 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:33:37.871 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:33:37.872 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:33:37.872 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:33:37.873 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:33:38.760 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2500377447 2023-10-20T09:33:38.761 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:33:38.762 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794418 2023-10-20T09:33:38.762 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:33:38.762 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2500377447 ]] 2023-10-20T09:33:38.762 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2500377447 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\5\0\0\3\7\7\4\4\7 ]] 2023-10-20T09:33:38.763 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:33:48.763 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:33:48.766 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:33:48.769 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:33:48.774 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:33:49.743 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2500377447 2023-10-20T09:33:49.743 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:33:49.744 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794429 2023-10-20T09:33:49.744 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:33:49.744 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2500377447 ]] 2023-10-20T09:33:49.744 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2500377447 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\5\0\0\3\7\7\4\4\7 ]] 2023-10-20T09:33:49.744 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:33:59.745 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:33:59.746 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:33:59.750 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:33:59.762 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:34:00.012 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.016+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.012 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.016+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc4fea00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.013 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.016+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.013 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.016+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc411480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.019 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.020+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.020 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.020+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc005c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.026 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.028+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.026 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.028+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc411e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.034 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.036+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.034 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.036+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcb33500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.042 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.044+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.042 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.044+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc46580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.048 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.052+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.049 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.052+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc47500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.056 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.060+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.056 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.060+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc47600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.062 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.064+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.063 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.064+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc47c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.069 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.072+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.069 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.072+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc46800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.086 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.088+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.087 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.088+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc46400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.093 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.096+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.093 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.096+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcb33780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.094 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.096+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.095 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.096+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc47f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.101 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.104+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.101 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.104+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bbbd6100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.107 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.108+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.107 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.108+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bbbd6300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.114 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.116+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.114 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.116+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bbbd6500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.120 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.124+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.120 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.124+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bbbd6700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.127 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.128+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.127 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.128+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bbbd6900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.134 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.136+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.134 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.136+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bbbd6b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.140 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.140+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.140 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:00.140+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bbbd6d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:00.719 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2500377447 2023-10-20T09:34:00.725 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:34:00.732 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794440 2023-10-20T09:34:00.732 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:34:00.732 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2500377447 ]] 2023-10-20T09:34:00.733 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2500377447 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\5\0\0\3\7\7\4\4\7 ]] 2023-10-20T09:34:00.733 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:34:10.738 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:34:10.738 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:34:10.738 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:34:10.742 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:34:11.566 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2500377447 2023-10-20T09:34:11.566 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:34:11.567 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794451 2023-10-20T09:34:11.567 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:34:11.568 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2500377447 ]] 2023-10-20T09:34:11.568 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2500377447 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\5\0\0\3\7\7\4\4\7 ]] 2023-10-20T09:34:11.568 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:34:14.453 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:34:14.456+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:34:14.456 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:34:14.460+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:34:14.456 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:34:14.460+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:34:14.470 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:34:14.472+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:34:21.582 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:34:21.584 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:34:21.584 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:34:21.591 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:34:22.470 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1398764105 2023-10-20T09:34:22.480 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:34:22.481 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794462 2023-10-20T09:34:22.482 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:34:22.482 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1398764105 ]] 2023-10-20T09:34:22.482 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1398764105 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\5\0\0\3\7\7\4\4\7 ]] 2023-10-20T09:34:22.482 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/1398764105 2023-10-20T09:34:25.058 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/1398764105 until 2023-10-20T10:34:24.246539+0000 (3600 sec) 2023-10-20T09:34:25.102 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:34:25.108 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/1398764105 2023-10-20T09:34:25.825 INFO:tasks.workunit.client.0.smithi150.stderr:listed 17 entries 2023-10-20T09:34:25.855 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/1398764105 2023-10-20T09:34:25.855 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:34:35.857 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:34:35.857 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:34:35.860 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:34:35.862 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:34:36.795 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1398764105 2023-10-20T09:34:36.796 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:34:36.796 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794476 2023-10-20T09:34:36.797 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:34:36.797 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1398764105 ]] 2023-10-20T09:34:36.797 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1398764105 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\3\9\8\7\6\4\1\0\5 ]] 2023-10-20T09:34:36.797 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:34:45.797 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:34:45.795+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:34:45.798 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:34:45.799+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T09:34:45.798 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:34:45.799+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:34:45.801 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:34:45.803+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab4f70 2023-10-20T09:34:45.801 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:34:45.803+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T09:34:45.801 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:34:45.803+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:34:45.802 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:34:45.803+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:34:45.802 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:34:45.803+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T09:34:45.802 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:34:45.803+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:34:45.802 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:34:45.803+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:34:46.806 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:34:46.811 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:34:46.816 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:34:46.825 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:34:47.856 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1398764105 2023-10-20T09:34:47.856 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:34:47.857 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794487 2023-10-20T09:34:47.857 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:34:47.857 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1398764105 ]] 2023-10-20T09:34:47.858 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1398764105 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\3\9\8\7\6\4\1\0\5 ]] 2023-10-20T09:34:47.858 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:34:57.860 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:34:57.870 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:34:57.880 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:34:57.883 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:34:58.820 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1398764105 2023-10-20T09:34:58.821 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:34:58.821 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794498 2023-10-20T09:34:58.822 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:34:58.822 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1398764105 ]] 2023-10-20T09:34:58.822 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1398764105 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\3\9\8\7\6\4\1\0\5 ]] 2023-10-20T09:34:58.822 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:34:59.999 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:59.999+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:34:59.999 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:34:59.999+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc210700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.009 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.011+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.009 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.011+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bccc2180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.010 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.011+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.010 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.011+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc47100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.016 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.015+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.016 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.015+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bccc2c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.026 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.027+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.027 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.027+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.027 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.027+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc47c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.027 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.027+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bbbd7480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.031 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.031+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.031 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.031+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bbbd6f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.038 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.039+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.038 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.039+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bbbd7080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.042 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.043+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.043 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.043+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bbbd7980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.046 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.047+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.047 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.047+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bbbd7800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.055 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.055+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.055 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.055+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bbbd7600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.062 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.063+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.062 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.063+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc2f7780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.069 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.071+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.070 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.071+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bbbd7c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.075 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.075+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.075 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.075+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bbbd7e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.081 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.083+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.081 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.083+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc54000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.087 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.087+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.087 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.087+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc54200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.093 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.095+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.093 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.095+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc54400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.099 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.099+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.099 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.099+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc54600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.106 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.107+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.106 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.107+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc54800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.112 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.111+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:00.112 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:00.111+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc54a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:08.823 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:35:08.830 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:35:08.831 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:35:08.831 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:35:09.793 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1398764105 2023-10-20T09:35:09.793 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:35:09.795 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794509 2023-10-20T09:35:09.795 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:35:09.795 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1398764105 ]] 2023-10-20T09:35:09.796 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1398764105 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\3\9\8\7\6\4\1\0\5 ]] 2023-10-20T09:35:09.796 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:35:19.797 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:35:19.797 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:35:19.797 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:35:19.810 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:35:20.682 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1398764105 2023-10-20T09:35:20.682 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:35:20.683 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794520 2023-10-20T09:35:20.683 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:35:20.683 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1398764105 ]] 2023-10-20T09:35:20.683 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1398764105 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\3\9\8\7\6\4\1\0\5 ]] 2023-10-20T09:35:20.683 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:35:30.684 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:35:30.685 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:35:30.687 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:35:30.692 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:35:31.670 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3786067858 2023-10-20T09:35:31.670 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:35:31.671 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794531 2023-10-20T09:35:31.671 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:35:31.671 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3786067858 ]] 2023-10-20T09:35:31.672 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3786067858 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\3\9\8\7\6\4\1\0\5 ]] 2023-10-20T09:35:31.672 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/3786067858 2023-10-20T09:35:34.098 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/3786067858 until 2023-10-20T10:35:33.333234+0000 (3600 sec) 2023-10-20T09:35:34.135 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:35:34.135 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/3786067858 2023-10-20T09:35:34.823 INFO:tasks.workunit.client.0.smithi150.stderr:listed 18 entries 2023-10-20T09:35:34.856 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/3786067858 2023-10-20T09:35:34.857 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:35:44.859 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:35:44.870 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:35:44.872 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:35:44.877 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:35:45.847 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3786067858 2023-10-20T09:35:45.847 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:35:45.848 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794545 2023-10-20T09:35:45.848 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:35:45.848 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3786067858 ]] 2023-10-20T09:35:45.849 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3786067858 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\8\6\0\6\7\8\5\8 ]] 2023-10-20T09:35:45.849 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:35:55.850 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:35:55.850 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:35:55.865 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:35:55.869 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:35:56.848 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3786067858 2023-10-20T09:35:56.848 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:35:56.849 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794556 2023-10-20T09:35:56.849 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:35:56.849 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3786067858 ]] 2023-10-20T09:35:56.849 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3786067858 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\8\6\0\6\7\8\5\8 ]] 2023-10-20T09:35:56.850 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:35:59.999 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:59.998+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:35:59.999 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:35:59.998+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bbbd6680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.005 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.006+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.005 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.006+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bb991280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.012 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.010+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.012 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.010+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc46d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.015 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.014+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.015 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.014+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bb990f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.019 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.022+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.020 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.022+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc46980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.040 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.042+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.040 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.042+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcb33e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.057 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.058+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.057 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.058+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc54a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.061 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.062+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.062 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.062+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc46880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.065 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.066+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.065 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.066+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc54b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.070 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.070+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.070 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.070+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc54c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.075 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.078+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.076 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.078+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc54e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.081 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.082+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.081 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.082+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc211080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.090 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.090+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.090 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.090+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc55180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.099 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.098+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.099 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.098+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc55380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.107 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.106+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.107 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.106+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc55580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.118 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.118+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.118 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.118+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc55780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.126 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.126+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.126 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.126+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc55980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.130 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.130+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.130 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.130+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc55b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.136 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.134+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.136 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.134+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc55d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.143 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.142+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:00.143 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:36:00.142+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc55f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:06.850 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:36:06.851 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:36:06.857 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:36:06.861 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:36:07.658 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3786067858 2023-10-20T09:36:07.661 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:36:07.665 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794567 2023-10-20T09:36:07.665 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:36:07.665 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3786067858 ]] 2023-10-20T09:36:07.665 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3786067858 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\8\6\0\6\7\8\5\8 ]] 2023-10-20T09:36:07.665 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:36:15.844 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:36:15.846+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:36:15.844 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:36:15.846+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:36:15.859 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:36:15.858+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:36:17.666 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:36:17.667 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:36:17.667 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:36:17.675 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:36:18.381 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3786067858 2023-10-20T09:36:18.381 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:36:18.382 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794578 2023-10-20T09:36:18.382 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:36:18.382 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3786067858 ]] 2023-10-20T09:36:18.382 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3786067858 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\8\6\0\6\7\8\5\8 ]] 2023-10-20T09:36:18.383 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:36:28.384 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:36:28.385 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:36:28.399 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:36:28.407 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:36:29.278 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3786067858 2023-10-20T09:36:29.278 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:36:29.279 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794589 2023-10-20T09:36:29.279 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:36:29.279 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3786067858 ]] 2023-10-20T09:36:29.279 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3786067858 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\8\6\0\6\7\8\5\8 ]] 2023-10-20T09:36:29.280 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:36:39.280 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:36:39.281 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:36:39.282 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:36:39.283 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:36:40.166 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/348031646 2023-10-20T09:36:40.166 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:36:40.167 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794600 2023-10-20T09:36:40.167 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:36:40.167 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/348031646 ]] 2023-10-20T09:36:40.167 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/348031646 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\8\6\0\6\7\8\5\8 ]] 2023-10-20T09:36:40.168 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/348031646 2023-10-20T09:36:42.691 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/348031646 until 2023-10-20T10:36:41.941097+0000 (3600 sec) 2023-10-20T09:36:42.736 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:36:42.744 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/348031646 2023-10-20T09:36:43.474 INFO:tasks.workunit.client.0.smithi150.stderr:listed 19 entries 2023-10-20T09:36:43.511 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/348031646 2023-10-20T09:36:43.511 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:36:44.464 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:36:44.457+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:36:44.464 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:36:44.461+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T09:36:44.465 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:36:44.461+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:36:44.472 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:36:44.473+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T09:36:44.472 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:36:44.473+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:36:44.472 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:36:44.473+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:36:44.473 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:36:44.473+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T09:36:44.473 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:36:44.473+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:36:44.473 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:36:44.473+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:36:53.513 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:36:53.513 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:36:53.516 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:36:53.530 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:36:54.380 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/348031646 2023-10-20T09:36:54.381 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:36:54.382 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794614 2023-10-20T09:36:54.382 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:36:54.382 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/348031646 ]] 2023-10-20T09:36:54.383 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/348031646 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\4\8\0\3\1\6\4\6 ]] 2023-10-20T09:36:54.383 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:36:59.999 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.001+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:36:59.999 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.001+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcb33700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.003 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.005+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.003 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.005+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc2f6380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.009 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.013+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.009 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.013+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc2f7100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.015 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.017+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.015 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.017+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc2f7080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.024 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.029+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.025 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.029+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bb991b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.030 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.029+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.031 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.033+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc4ffe00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.034 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.037+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.034 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.037+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc55b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.041 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.045+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.041 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.045+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc55300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.048 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.049+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.048 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.049+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc6b1800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.054 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.057+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.055 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.057+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc47900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.061 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.065+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.061 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.065+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bbbd6200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.068 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.069+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.068 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.069+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc55b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.075 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.077+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.076 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.077+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bce06000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.084 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.085+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.085 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.085+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bce06300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.096 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.097+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.097 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.097+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bce06100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.104 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.105+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.104 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.105+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bce06980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.110 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.113+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.111 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.113+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bce06b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.118 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.121+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.118 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.121+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bce06d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.124 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.129+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.125 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.129+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bce07780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.132 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.133+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:00.132 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:37:00.133+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bce07600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:37:04.398 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:37:04.403 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:37:04.405 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:37:04.413 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:37:05.342 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/348031646 2023-10-20T09:37:05.350 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:37:05.351 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794625 2023-10-20T09:37:05.351 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:37:05.352 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/348031646 ]] 2023-10-20T09:37:05.352 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/348031646 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\4\8\0\3\1\6\4\6 ]] 2023-10-20T09:37:05.352 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:37:15.355 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:37:15.355 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:37:15.356 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:37:15.359 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:37:16.444 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/348031646 2023-10-20T09:37:16.445 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:37:16.446 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794636 2023-10-20T09:37:16.447 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:37:16.447 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/348031646 ]] 2023-10-20T09:37:16.447 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/348031646 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\4\8\0\3\1\6\4\6 ]] 2023-10-20T09:37:16.448 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:37:26.447 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:37:26.448 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:37:26.448 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:37:26.466 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:37:27.368 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/348031646 2023-10-20T09:37:27.368 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:37:27.369 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794647 2023-10-20T09:37:27.369 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:37:27.369 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/348031646 ]] 2023-10-20T09:37:27.369 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/348031646 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\4\8\0\3\1\6\4\6 ]] 2023-10-20T09:37:27.369 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:37:34.109 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:37:34.112+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:37:34.122 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:37:34.124+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T09:37:34.122 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:37:34.124+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:37:34.123 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:37:34.124+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:37:37.371 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:37:37.371 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:37:37.376 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:37:37.377 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:37:38.262 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/348031646 2023-10-20T09:37:38.262 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:37:38.263 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794658 2023-10-20T09:37:38.263 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:37:38.263 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/348031646 ]] 2023-10-20T09:37:38.264 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/348031646 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\4\8\0\3\1\6\4\6 ]] 2023-10-20T09:37:38.264 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:37:39.726 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:37:39.728+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:37:39.726 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:37:39.728+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T09:37:39.727 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:37:39.728+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:37:39.736 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:37:39.736+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T09:37:39.736 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:37:39.736+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:37:39.737 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:37:39.736+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:37:39.737 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:37:39.736+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T09:37:39.737 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:37:39.736+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:37:39.737 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:37:39.740+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:37:46.238 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:37:46.236+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:37:46.239 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:37:46.236+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:37:48.265 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:37:48.265 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:37:48.265 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:37:48.271 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:37:49.257 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2299851903 2023-10-20T09:37:49.258 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:37:49.260 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794669 2023-10-20T09:37:49.260 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:37:49.260 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2299851903 ]] 2023-10-20T09:37:49.260 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2299851903 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\4\8\0\3\1\6\4\6 ]] 2023-10-20T09:37:49.260 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/2299851903 2023-10-20T09:37:51.139 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/2299851903 until 2023-10-20T10:37:50.323961+0000 (3600 sec) 2023-10-20T09:37:51.183 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:37:51.184 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/2299851903 2023-10-20T09:37:51.937 INFO:tasks.workunit.client.0.smithi150.stderr:listed 20 entries 2023-10-20T09:37:51.967 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/2299851903 2023-10-20T09:37:51.967 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:38:00.008 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.004+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.009 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.004+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.009 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.004+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bcada980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.009 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.004+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bcc47680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.009 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.008+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.010 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.008+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bbbd7800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.010 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.012+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.010 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.012+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc920e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.019 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.020+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.019 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.020+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bbbd7000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.030 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.032+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.030 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.032+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bbbd7f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.034 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.036+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.034 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.036+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bce07280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.039 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.040+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.039 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.040+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bce07100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.043 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.044+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.043 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.044+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bce06280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.047 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.048+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.047 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.048+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bce06000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.052 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.056+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.052 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.056+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bce07480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.063 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.064+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.064 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.064+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bbbd7e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.068 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.072+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.069 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.072+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bce06100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.074 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.076+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.075 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.076+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bce06980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.084 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.088+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.084 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.088+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bce06b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.089 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.092+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.089 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.092+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bce06d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.093 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.096+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.094 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.096+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bce07780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.099 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.100+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.099 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.100+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bce07900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.104 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.108+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.104 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.108+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bce07b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.110 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.112+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:00.111 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:38:00.112+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bce07d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:38:01.970 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:38:01.970 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:38:01.975 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:38:01.976 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:38:02.818 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2299851903 2023-10-20T09:38:02.818 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:38:02.819 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794682 2023-10-20T09:38:02.819 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:38:02.819 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2299851903 ]] 2023-10-20T09:38:02.820 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2299851903 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\2\9\9\8\5\1\9\0\3 ]] 2023-10-20T09:38:02.820 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:38:12.823 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:38:12.827 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:38:12.837 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:38:12.843 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:38:13.701 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2299851903 2023-10-20T09:38:13.701 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:38:13.702 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794693 2023-10-20T09:38:13.702 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:38:13.702 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2299851903 ]] 2023-10-20T09:38:13.702 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2299851903 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\2\9\9\8\5\1\9\0\3 ]] 2023-10-20T09:38:13.702 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:38:23.708 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:38:23.708 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:38:23.709 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:38:23.709 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:38:24.092 INFO:tasks.workunit.client.0.smithi150.stderr:2023-10-20T09:38:24.095+0000 7fb7aeffd640 0 --2- 172.21.15.150:0/1405536013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7b0071140 0x7fb7b011c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-20T09:38:24.783 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2299851903 2023-10-20T09:38:24.784 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:38:24.784 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794704 2023-10-20T09:38:24.784 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:38:24.785 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2299851903 ]] 2023-10-20T09:38:24.785 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2299851903 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\2\9\9\8\5\1\9\0\3 ]] 2023-10-20T09:38:24.785 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:38:32.308 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:38:32.311+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:38:32.308 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:38:32.311+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:38:32.315 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:38:32.319+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:38:34.786 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:38:34.786 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:38:34.789 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:38:34.793 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:38:35.758 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2299851903 2023-10-20T09:38:35.758 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:38:35.759 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794715 2023-10-20T09:38:35.759 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:38:35.759 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2299851903 ]] 2023-10-20T09:38:35.760 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2299851903 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\2\9\9\8\5\1\9\0\3 ]] 2023-10-20T09:38:35.760 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:38:45.761 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:38:45.761 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:38:45.780 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:38:45.792 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:38:46.447 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:38:46.447+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:38:46.447 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:38:46.447+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:38:46.466 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:38:46.467+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:38:46.792 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2299851903 2023-10-20T09:38:46.792 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:38:46.793 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794726 2023-10-20T09:38:46.793 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:38:46.793 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2299851903 ]] 2023-10-20T09:38:46.793 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2299851903 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\2\9\9\8\5\1\9\0\3 ]] 2023-10-20T09:38:46.794 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:38:50.906 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:38:50.907+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:38:50.909 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:38:50.907+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:38:56.795 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:38:56.796 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:38:56.803 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:38:56.811 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:38:57.696 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/998694798 2023-10-20T09:38:57.697 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:38:57.697 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794737 2023-10-20T09:38:57.698 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:38:57.698 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/998694798 ]] 2023-10-20T09:38:57.698 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/998694798 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\2\9\9\8\5\1\9\0\3 ]] 2023-10-20T09:38:57.699 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/998694798 2023-10-20T09:38:59.713 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/998694798 until 2023-10-20T10:38:58.962855+0000 (3600 sec) 2023-10-20T09:38:59.757 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/998694798 2023-10-20T09:38:59.763 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:39:00.000 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.003+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.001 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.003+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc005380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.002 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.003+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.002 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.003+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bb991f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.006 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.007+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.006 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.007+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc2f7080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.026 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.027+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.026 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.027+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bb991b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.029 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.031+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.030 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.031+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bbbd7780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.033 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.035+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.033 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.035+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc54000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.045 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.047+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.046 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.047+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcb33f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.052 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.055+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.053 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.055+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc54680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.058 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.059+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.059 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.059+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc6b1700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.064 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.067+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.065 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.067+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc46b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.070 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.071+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.071 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.071+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcb32780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.077 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.079+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.077 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.079+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcb32380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.083 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.083+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.084 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.083+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bce07f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.090 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.091+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.090 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.091+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcdf6080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.096 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.099+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.096 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.099+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcdf6280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.102 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.103+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.102 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.103+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcdf6480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.108 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.111+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.109 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.111+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcdf6e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.115 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.115+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.115 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.115+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcdf6d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.123 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.123+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.124 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.123+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcdf6b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.127 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.127+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.127 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:39:00.127+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcdf6900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:39:00.607 INFO:tasks.workunit.client.0.smithi150.stderr:listed 21 entries 2023-10-20T09:39:00.638 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/998694798 2023-10-20T09:39:00.639 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:39:10.642 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:39:10.643 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:39:10.643 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:39:10.643 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:39:11.497 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/998694798 2023-10-20T09:39:11.497 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:39:11.498 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794751 2023-10-20T09:39:11.498 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:39:11.498 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/998694798 ]] 2023-10-20T09:39:11.498 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/998694798 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\9\9\8\6\9\4\7\9\8 ]] 2023-10-20T09:39:11.499 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:39:21.500 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:39:21.503 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:39:21.505 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:39:21.510 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:39:22.400 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/998694798 2023-10-20T09:39:22.400 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:39:22.401 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794762 2023-10-20T09:39:22.401 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:39:22.401 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/998694798 ]] 2023-10-20T09:39:22.401 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/998694798 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\9\9\8\6\9\4\7\9\8 ]] 2023-10-20T09:39:22.402 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:39:32.403 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:39:32.403 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:39:32.405 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:39:32.426 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:39:33.426 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/998694798 2023-10-20T09:39:33.427 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:39:33.427 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794773 2023-10-20T09:39:33.428 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:39:33.428 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/998694798 ]] 2023-10-20T09:39:33.428 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/998694798 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\9\9\8\6\9\4\7\9\8 ]] 2023-10-20T09:39:33.428 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:39:35.859 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:39:35.862+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:39:43.435 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:39:43.438 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:39:43.440 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:39:43.461 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:39:44.318 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/998694798 2023-10-20T09:39:44.319 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:39:44.320 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794784 2023-10-20T09:39:44.321 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:39:44.321 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/998694798 ]] 2023-10-20T09:39:44.321 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/998694798 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\9\9\8\6\9\4\7\9\8 ]] 2023-10-20T09:39:44.321 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:39:54.325 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:39:54.332 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:39:54.338 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:39:54.341 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:39:55.292 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/998694798 2023-10-20T09:39:55.292 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:39:55.293 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794795 2023-10-20T09:39:55.293 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:39:55.293 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/998694798 ]] 2023-10-20T09:39:55.293 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/998694798 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\9\9\8\6\9\4\7\9\8 ]] 2023-10-20T09:39:55.293 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:40:05.294 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:40:05.298 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:40:05.299 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:40:05.299 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:40:06.254 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1510323101 2023-10-20T09:40:06.273 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:40:06.274 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794806 2023-10-20T09:40:06.275 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:40:06.275 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1510323101 ]] 2023-10-20T09:40:06.275 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1510323101 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\9\9\8\6\9\4\7\9\8 ]] 2023-10-20T09:40:06.275 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/1510323101 2023-10-20T09:40:08.743 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/1510323101 until 2023-10-20T10:40:07.983139+0000 (3600 sec) 2023-10-20T09:40:08.788 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:40:08.789 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/1510323101 2023-10-20T09:40:09.624 INFO:tasks.workunit.client.0.smithi150.stderr:listed 22 entries 2023-10-20T09:40:09.659 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/1510323101 2023-10-20T09:40:09.659 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:40:11.731 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:40:11.734+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:40:19.675 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:40:19.679 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:40:19.682 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:40:19.684 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:40:20.649 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1510323101 2023-10-20T09:40:20.650 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:40:20.650 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794820 2023-10-20T09:40:20.651 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:40:20.651 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1510323101 ]] 2023-10-20T09:40:20.651 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1510323101 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\5\1\0\3\2\3\1\0\1 ]] 2023-10-20T09:40:20.651 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:40:30.485 INFO:tasks.ceph.osd.2.smithi150.stderr:2023-10-20T09:40:30.486+0000 7f12576b6640 -1 reset not still connected to 0x5635c0956d00 2023-10-20T09:40:30.485 INFO:tasks.ceph.osd.2.smithi150.stderr:2023-10-20T09:40:30.486+0000 7f12576b6640 -1 reset not still connected to 0x5635c0e90410 2023-10-20T09:40:30.485 INFO:tasks.ceph.osd.2.smithi150.stderr:2023-10-20T09:40:30.486+0000 7f12576b6640 -1 reset not still connected to 0x5635c13f41a0 2023-10-20T09:40:30.486 INFO:tasks.ceph.osd.2.smithi150.stderr:2023-10-20T09:40:30.486+0000 7f12576b6640 -1 reset not still connected to 0x5635c19f9040 2023-10-20T09:40:30.654 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:40:30.655 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:40:30.655 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:40:30.678 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:40:31.615 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1510323101 2023-10-20T09:40:31.617 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:40:31.618 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794831 2023-10-20T09:40:31.618 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:40:31.618 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1510323101 ]] 2023-10-20T09:40:31.618 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1510323101 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\5\1\0\3\2\3\1\0\1 ]] 2023-10-20T09:40:31.619 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:40:35.084 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:40:35.086+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T09:40:35.084 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:40:35.086+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:40:35.084 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:40:35.086+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T09:40:35.085 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:40:35.086+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:40:35.086 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:40:35.086+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:40:41.624 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:40:41.625 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:40:41.633 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:40:41.635 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:40:42.504 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1510323101 2023-10-20T09:40:42.504 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:40:42.505 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794842 2023-10-20T09:40:42.505 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:40:42.505 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1510323101 ]] 2023-10-20T09:40:42.506 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1510323101 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\5\1\0\3\2\3\1\0\1 ]] 2023-10-20T09:40:42.506 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:40:52.506 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:40:52.506 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:40:52.515 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:40:52.516 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:40:53.313 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1510323101 2023-10-20T09:40:53.314 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:40:53.314 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794853 2023-10-20T09:40:53.315 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:40:53.315 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1510323101 ]] 2023-10-20T09:40:53.315 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1510323101 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\5\1\0\3\2\3\1\0\1 ]] 2023-10-20T09:40:53.315 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:41:00.031 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.033+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.032 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.033+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.032 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.033+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.032 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.033+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.033 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.033+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcdf7380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.033 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.033+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcdf7580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.033 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.033+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcdf7780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.033 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.033+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcdf7980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.039 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.041+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.039 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.041+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcdf7b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.042 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.041+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.042 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.041+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcdf7d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.060 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.061+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.061 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.061+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcdf7f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.068 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.069+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.068 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.069+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc918180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.075 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.073+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.075 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.073+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc918380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.083 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.081+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.083 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.081+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc918580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.084 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.085+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.085 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.085+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc918c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.087 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.089+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.087 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.089+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc918b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.090 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.089+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.090 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.089+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc918900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.093 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.093+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.093 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.093+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc918d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.095 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.097+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.095 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.097+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc918f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.097 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.097+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.098 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.097+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc919180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.101 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.101+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.101 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.101+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc919380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.104 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.105+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.104 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.105+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc919580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.109 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.109+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.109 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.109+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc919780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.114 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.113+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:00.114 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:41:00.113+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc919980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:41:03.316 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:41:03.333 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:41:03.338 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:41:03.351 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:41:04.375 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1510323101 2023-10-20T09:41:04.376 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:41:04.380 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794864 2023-10-20T09:41:04.380 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:41:04.380 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1510323101 ]] 2023-10-20T09:41:04.381 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1510323101 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\5\1\0\3\2\3\1\0\1 ]] 2023-10-20T09:41:04.381 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:41:14.383 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:41:14.383 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:41:14.391 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:41:14.401 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:41:15.381 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2624331567 2023-10-20T09:41:15.381 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:41:15.382 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794875 2023-10-20T09:41:15.382 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:41:15.382 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2624331567 ]] 2023-10-20T09:41:15.382 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2624331567 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\5\1\0\3\2\3\1\0\1 ]] 2023-10-20T09:41:15.383 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/2624331567 2023-10-20T09:41:17.753 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/2624331567 until 2023-10-20T10:41:17.070962+0000 (3600 sec) 2023-10-20T09:41:17.811 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:41:17.816 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/2624331567 2023-10-20T09:41:18.597 INFO:tasks.workunit.client.0.smithi150.stderr:listed 23 entries 2023-10-20T09:41:18.634 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/2624331567 2023-10-20T09:41:18.635 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:41:28.647 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:41:28.649 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:41:28.650 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:41:28.651 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:41:29.587 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2624331567 2023-10-20T09:41:29.587 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:41:29.588 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794889 2023-10-20T09:41:29.588 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:41:29.588 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2624331567 ]] 2023-10-20T09:41:29.588 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2624331567 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\6\2\4\3\3\1\5\6\7 ]] 2023-10-20T09:41:29.588 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:41:37.050 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:41:37.052+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:41:37.051 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:41:37.052+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:41:37.051 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:41:37.052+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:41:39.590 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:41:39.594 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:41:39.596 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:41:39.596 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:41:40.522 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2624331567 2023-10-20T09:41:40.522 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:41:40.523 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794900 2023-10-20T09:41:40.523 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:41:40.523 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2624331567 ]] 2023-10-20T09:41:40.523 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2624331567 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\6\2\4\3\3\1\5\6\7 ]] 2023-10-20T09:41:40.524 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:41:50.524 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:41:50.531 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:41:50.539 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:41:50.544 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:41:51.457 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2624331567 2023-10-20T09:41:51.458 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:41:51.458 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794911 2023-10-20T09:41:51.459 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:41:51.459 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2624331567 ]] 2023-10-20T09:41:51.459 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2624331567 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\6\2\4\3\3\1\5\6\7 ]] 2023-10-20T09:41:51.459 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:42:00.015 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.015+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.016 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.015+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bbbd6d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.019 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.019+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.019 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.019+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcc54900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.023 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.023+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.023 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.023+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcc54880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.030 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.031+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.030 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.031+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcc55a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.037 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.039+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.037 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.039+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcdf7080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.060 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.059+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.060 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.059+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcdf6800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.065 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.067+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.065 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.067+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcdf7f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.070 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.071+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.070 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.071+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcdf7e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.076 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.079+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.076 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.079+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bce06380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.085 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.087+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.085 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.087+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bce06480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.098 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.091+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.098 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.095+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcdf6980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.099 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.099+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.100 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.099+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc95bd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.101 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.103+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.102 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.103+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bce07980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.120 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.123+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.121 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.123+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bce07a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.135 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.135+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.135 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.135+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bce06080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.139 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.139+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.139 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.139+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc918100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.143 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.143+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.143 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.143+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc918500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.146 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.147+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.147 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.147+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc005c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.154 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.155+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.155 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.155+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bce06600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.161 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.163+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:00.161 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:42:00.163+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc919e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:42:01.480 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:42:01.481 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:42:01.491 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:42:01.504 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:42:02.655 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2624331567 2023-10-20T09:42:02.696 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:42:02.697 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794922 2023-10-20T09:42:02.697 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:42:02.697 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2624331567 ]] 2023-10-20T09:42:02.698 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2624331567 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\6\2\4\3\3\1\5\6\7 ]] 2023-10-20T09:42:02.698 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:42:07.897 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:42:07.899+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:42:07.903 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:42:07.903+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:42:11.505 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:42:11.503+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:42:12.668 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:42:12.668 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:42:12.668 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:42:12.668 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:42:13.617 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2624331567 2023-10-20T09:42:13.618 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:42:13.618 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794933 2023-10-20T09:42:13.618 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:42:13.619 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2624331567 ]] 2023-10-20T09:42:13.619 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2624331567 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\6\2\4\3\3\1\5\6\7 ]] 2023-10-20T09:42:13.619 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:42:23.620 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:42:23.621 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:42:23.621 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:42:23.642 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:42:24.624 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/2624331567 2023-10-20T09:42:24.624 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:42:24.625 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794944 2023-10-20T09:42:24.625 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:42:24.625 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/2624331567 ]] 2023-10-20T09:42:24.625 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/2624331567 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\6\2\4\3\3\1\5\6\7 ]] 2023-10-20T09:42:24.625 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:42:34.628 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:42:34.628 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:42:34.628 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:42:34.634 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:42:35.524 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3221254932 2023-10-20T09:42:35.524 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:42:35.525 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794955 2023-10-20T09:42:35.525 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:42:35.525 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3221254932 ]] 2023-10-20T09:42:35.526 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3221254932 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\6\2\4\3\3\1\5\6\7 ]] 2023-10-20T09:42:35.526 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/3221254932 2023-10-20T09:42:37.757 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/3221254932 until 2023-10-20T10:42:36.956804+0000 (3600 sec) 2023-10-20T09:42:37.790 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/3221254932 2023-10-20T09:42:37.791 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:42:38.474 INFO:tasks.workunit.client.0.smithi150.stderr:listed 24 entries 2023-10-20T09:42:38.535 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/3221254932 2023-10-20T09:42:38.535 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:42:48.544 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:42:48.547 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:42:48.556 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:42:48.559 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:42:49.439 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3221254932 2023-10-20T09:42:49.440 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:42:49.440 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794969 2023-10-20T09:42:49.440 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:42:49.441 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3221254932 ]] 2023-10-20T09:42:49.441 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3221254932 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\2\2\1\2\5\4\9\3\2 ]] 2023-10-20T09:42:49.441 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:42:59.452 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:42:59.452 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:42:59.453 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:42:59.468 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:43:00.009 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.010+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.010 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.010+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.010 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.010+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.010 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.010+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bcc46100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.010 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.010+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bccc3780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.011 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.010+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bcadb900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.011 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.010+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.011 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.010+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bccc3a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.014 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.014+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.014 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.014+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bcadaf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.025 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.026+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.025 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.026+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bcdf6700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.029 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.030+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.029 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.030+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bbbd6680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.033 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.034+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.034 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.034+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bbbd7b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.044 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.042+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.044 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.046+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bcdf6d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.066 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.066+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.066 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.066+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bce06c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.073 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.070+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.074 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.070+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bcc55500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.074 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.074+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.074 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.074+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bbbd7f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.076 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.078+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.076 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.078+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc919a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.080 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.082+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.081 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.082+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc919c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.086 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.086+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.087 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.086+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bcf5c080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.096 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.098+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.096 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.098+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bcf5c280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.103 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.102+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.103 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.102+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bcf5c480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.119 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.118+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.119 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.118+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bcf5ce80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.126 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.126+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.126 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.126+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bcf5cd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.133 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.134+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.133 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:43:00.134+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bcf5cb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:43:00.432 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3221254932 2023-10-20T09:43:00.432 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:43:00.433 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794980 2023-10-20T09:43:00.433 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:43:00.433 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3221254932 ]] 2023-10-20T09:43:00.433 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3221254932 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\2\2\1\2\5\4\9\3\2 ]] 2023-10-20T09:43:00.434 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:43:09.447 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:43:09.446+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:43:09.448 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:43:09.446+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:43:09.980 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:43:09.970+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:43:09.990 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:43:09.990+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T09:43:09.990 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:43:09.990+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:43:09.992 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:43:09.990+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab4f70 2023-10-20T09:43:09.992 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:43:09.990+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T09:43:09.992 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:43:09.990+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:43:09.992 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:43:09.990+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:43:09.993 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:43:09.990+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T09:43:09.993 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:43:09.990+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:43:09.993 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:43:09.990+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:43:09.993 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:43:09.990+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2750 2023-10-20T09:43:09.993 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:43:09.990+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2820 2023-10-20T09:43:09.994 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:43:09.990+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:43:10.446 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:43:10.446 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:43:10.446 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:43:10.468 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:43:11.285 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3221254932 2023-10-20T09:43:11.286 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:43:11.286 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697794991 2023-10-20T09:43:11.286 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:43:11.287 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3221254932 ]] 2023-10-20T09:43:11.287 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3221254932 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\2\2\1\2\5\4\9\3\2 ]] 2023-10-20T09:43:11.287 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:43:21.288 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:43:21.288 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:43:21.289 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:43:21.289 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:43:22.196 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3221254932 2023-10-20T09:43:22.197 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:43:22.197 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795002 2023-10-20T09:43:22.198 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:43:22.198 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3221254932 ]] 2023-10-20T09:43:22.198 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3221254932 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\2\2\1\2\5\4\9\3\2 ]] 2023-10-20T09:43:22.198 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:43:26.155 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:43:26.153+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:43:26.166 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:43:26.161+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:43:32.199 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:43:32.207 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:43:32.209 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:43:32.237 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:43:33.148 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3221254932 2023-10-20T09:43:33.148 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:43:33.149 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795013 2023-10-20T09:43:33.149 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:43:33.149 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3221254932 ]] 2023-10-20T09:43:33.150 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3221254932 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\2\2\1\2\5\4\9\3\2 ]] 2023-10-20T09:43:33.150 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:43:43.163 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:43:43.163 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:43:43.163 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:43:43.196 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:43:44.189 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/4240815768 2023-10-20T09:43:44.189 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:43:44.190 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795024 2023-10-20T09:43:44.190 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:43:44.190 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/4240815768 ]] 2023-10-20T09:43:44.190 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/4240815768 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\2\2\1\2\5\4\9\3\2 ]] 2023-10-20T09:43:44.190 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/4240815768 2023-10-20T09:43:46.270 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/4240815768 until 2023-10-20T10:43:45.489951+0000 (3600 sec) 2023-10-20T09:43:46.305 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/4240815768 2023-10-20T09:43:46.314 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:43:47.010 INFO:tasks.workunit.client.0.smithi150.stderr:listed 25 entries 2023-10-20T09:43:47.041 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/4240815768 2023-10-20T09:43:47.042 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:43:57.043 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:43:57.044 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:43:57.044 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:43:57.045 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:43:57.993 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/4240815768 2023-10-20T09:43:57.998 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:43:58.000 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795038 2023-10-20T09:43:58.000 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:43:58.001 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/4240815768 ]] 2023-10-20T09:43:58.001 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/4240815768 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\4\2\4\0\8\1\5\7\6\8 ]] 2023-10-20T09:43:58.001 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:44:00.000 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.000+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.001 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.000+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bb7b5d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.003 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.004+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.003 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.004+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bbbd7d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.013 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.016+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.013 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.016+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bce07580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.028 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.032+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.028 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.032+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcb32e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.030 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.032+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.030 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.032+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc918d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.036 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.036+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.036 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.036+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bb990f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.043 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.044+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.043 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.044+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcf5d280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.049 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.052+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.049 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.052+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcf5ca80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.056 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.056+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.056 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.056+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcf5d800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.062 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.064+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.062 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.064+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcf5d680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.067 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.068+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.068 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.068+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcf5d480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.072 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.072+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.072 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.072+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc004000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.076 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.080+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.076 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.080+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcf5da80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.080 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.080+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.080 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.080+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcf5dc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.087 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.088+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.088 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.088+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcf5de80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.093 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.096+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.094 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.096+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcdb0080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.100 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.100+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.100 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.100+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcdb0280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.106 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.108+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.106 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.108+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcdb0480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.112 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.112+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.113 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.112+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcdb0e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.119 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.120+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:00.119 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:44:00.120+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcdb0d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:44:07.336 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:44:07.332+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:44:08.006 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:44:08.007 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:44:08.009 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:44:08.013 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:44:08.835 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/4240815768 2023-10-20T09:44:08.835 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:44:08.835 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795048 2023-10-20T09:44:08.836 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:44:08.836 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/4240815768 ]] 2023-10-20T09:44:08.836 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/4240815768 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\4\2\4\0\8\1\5\7\6\8 ]] 2023-10-20T09:44:08.836 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:44:09.065 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:44:09.068+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:44:09.065 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:44:09.068+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:44:18.837 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:44:18.838 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:44:18.838 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:44:18.843 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:44:19.864 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/4240815768 2023-10-20T09:44:19.865 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:44:19.866 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795059 2023-10-20T09:44:19.866 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:44:19.866 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/4240815768 ]] 2023-10-20T09:44:19.866 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/4240815768 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\4\2\4\0\8\1\5\7\6\8 ]] 2023-10-20T09:44:19.867 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:44:29.868 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:44:29.869 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:44:29.869 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:44:29.885 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:44:30.785 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/4240815768 2023-10-20T09:44:30.785 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:44:30.786 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795070 2023-10-20T09:44:30.786 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:44:30.787 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/4240815768 ]] 2023-10-20T09:44:30.787 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/4240815768 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\4\2\4\0\8\1\5\7\6\8 ]] 2023-10-20T09:44:30.787 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:44:40.788 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:44:40.788 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:44:40.791 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:44:40.798 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:44:41.682 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/4240815768 2023-10-20T09:44:41.682 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:44:41.683 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795081 2023-10-20T09:44:41.683 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:44:41.684 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/4240815768 ]] 2023-10-20T09:44:41.684 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/4240815768 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\4\2\4\0\8\1\5\7\6\8 ]] 2023-10-20T09:44:41.684 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:44:51.685 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:44:51.686 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:44:51.686 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:44:51.690 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:44:52.487 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/436164353 2023-10-20T09:44:52.487 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:44:52.488 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795092 2023-10-20T09:44:52.488 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:44:52.489 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/436164353 ]] 2023-10-20T09:44:52.489 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/436164353 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\4\2\4\0\8\1\5\7\6\8 ]] 2023-10-20T09:44:52.489 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/436164353 2023-10-20T09:44:54.787 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/436164353 until 2023-10-20T10:44:54.003381+0000 (3600 sec) 2023-10-20T09:44:54.830 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:44:54.832 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/436164353 2023-10-20T09:44:55.453 INFO:tasks.workunit.client.0.smithi150.stderr:listed 26 entries 2023-10-20T09:44:55.478 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/436164353 2023-10-20T09:44:55.478 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:44:57.091 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:44:57.091+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:44:57.091 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:44:57.091+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T09:44:57.091 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:44:57.091+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:44:57.106 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:44:57.107+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab4f70 2023-10-20T09:44:57.106 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:44:57.107+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T09:44:57.106 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:44:57.107+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:44:57.107 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:44:57.107+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:44:57.107 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:44:57.107+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T09:44:57.107 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:44:57.107+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:44:57.107 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:44:57.107+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:44:57.108 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:44:57.107+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:44:57.108 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:44:57.107+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T09:44:57.108 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:44:57.107+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:44:57.108 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:44:57.107+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:44:57.108 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:44:57.107+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:45:00.013 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.007+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.013 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.007+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc073a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.013 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.011+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.013 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.011+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bb8d3200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.020 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.015+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.021 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.015+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bbbd6f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.023 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.023+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.023 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.023+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bb8d2680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.031 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.031+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.031 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.031+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bccc3780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.045 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.043+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.045 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.043+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc55a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.052 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.051+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.052 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.055+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcdf7600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.056 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.059+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.057 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.059+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcdf7800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.061 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.063+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.061 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.063+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc47980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.073 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.075+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.073 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.075+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc46e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.078 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.079+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.078 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.079+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcdf6800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.083 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.083+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.084 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.083+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bcdf6080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.089 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.091+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.090 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.091+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc410300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.095 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.095+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.095 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.095+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bcb32200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.105 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.107+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.105 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.107+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc920e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.110 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.111+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.110 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.111+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc918280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.115 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.115+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.115 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.115+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc918980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.120 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.119+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.120 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.119+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bc918c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.127 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.127+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.128 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.127+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bcadbe00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.139 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.139+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:00.140 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:45:00.139+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bcadb480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:45:05.480 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:45:05.481 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:45:05.481 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:45:05.492 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:45:06.379 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/436164353 2023-10-20T09:45:06.380 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:45:06.389 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795106 2023-10-20T09:45:06.389 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:45:06.389 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/436164353 ]] 2023-10-20T09:45:06.389 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/436164353 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\4\3\6\1\6\4\3\5\3 ]] 2023-10-20T09:45:06.390 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:45:12.670 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:45:12.667+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:45:12.670 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:45:12.667+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:45:16.394 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:45:16.395 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:45:16.395 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:45:16.404 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:45:17.279 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/436164353 2023-10-20T09:45:17.280 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:45:17.281 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795117 2023-10-20T09:45:17.281 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:45:17.281 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/436164353 ]] 2023-10-20T09:45:17.282 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/436164353 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\4\3\6\1\6\4\3\5\3 ]] 2023-10-20T09:45:17.282 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:45:27.282 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:45:27.283 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:45:27.283 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:45:27.283 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:45:28.256 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/436164353 2023-10-20T09:45:28.259 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:45:28.260 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795128 2023-10-20T09:45:28.260 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:45:28.261 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/436164353 ]] 2023-10-20T09:45:28.261 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/436164353 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\4\3\6\1\6\4\3\5\3 ]] 2023-10-20T09:45:28.261 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:45:34.131 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:45:34.130+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:45:34.131 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:45:34.130+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:45:34.131 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:45:34.130+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:45:37.311 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:45:37.310+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:45:37.312 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:45:37.310+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T09:45:37.312 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:45:37.310+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:45:37.312 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:45:37.310+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab4f70 2023-10-20T09:45:37.312 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:45:37.310+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T09:45:37.313 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:45:37.310+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:45:37.313 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:45:37.310+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:45:37.313 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:45:37.310+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T09:45:37.313 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:45:37.310+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:45:37.313 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:45:37.310+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:45:37.314 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:45:37.310+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2750 2023-10-20T09:45:37.314 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:45:37.310+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2820 2023-10-20T09:45:38.268 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:45:38.278 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:45:38.280 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:45:38.281 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:45:39.068 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/436164353 2023-10-20T09:45:39.068 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:45:39.069 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795139 2023-10-20T09:45:39.069 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:45:39.069 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/436164353 ]] 2023-10-20T09:45:39.070 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/436164353 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\4\3\6\1\6\4\3\5\3 ]] 2023-10-20T09:45:39.070 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:45:49.072 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:45:49.072 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:45:49.076 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:45:49.085 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:45:50.038 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/436164353 2023-10-20T09:45:50.039 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:45:50.040 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795150 2023-10-20T09:45:50.040 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:45:50.040 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/436164353 ]] 2023-10-20T09:45:50.041 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/436164353 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\4\3\6\1\6\4\3\5\3 ]] 2023-10-20T09:45:50.041 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:46:00.049 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:46:00.051 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:46:00.065 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:46:00.073 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:46:00.960 INFO:tasks.ceph.osd.2.smithi150.stderr:2023-10-20T09:46:00.958+0000 7f12576b6640 -1 reset not still connected to 0x5635dca3a000 2023-10-20T09:46:00.960 INFO:tasks.ceph.osd.2.smithi150.stderr:2023-10-20T09:46:00.958+0000 7f12576b6640 -1 reset not still connected to 0x5635f0455110 2023-10-20T09:46:01.008 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/389382636 2023-10-20T09:46:01.011 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:46:01.012 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795161 2023-10-20T09:46:01.012 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:46:01.013 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/389382636 ]] 2023-10-20T09:46:01.013 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/389382636 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\4\3\6\1\6\4\3\5\3 ]] 2023-10-20T09:46:01.013 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/389382636 2023-10-20T09:46:01.819 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.818+0000 7f28a3c2f640 -1 librbd::managed_lock::BreakRequest: 0x5610bc3652c0 handle_get_watchers: failed to retrieve watchers: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:01.819 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.818+0000 7f28a3c2f640 -1 librbd::managed_lock::AcquireRequest: 0x5610bcf6dc20 handle_break_lock: failed to break lock : (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:01.819 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.818+0000 7f28a3c2f640 -1 librbd::ManagedLock: 0x5610bb85ba18 handle_acquire_lock: failed to acquire exclusive lock: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:01.820 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.818+0000 7f28a3c2f640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5610bb8e0680 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:01.820 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.818+0000 7f28a3c2f640 -1 librbd::io::AioCompletion: 0x5610bd1eedc0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:01.822 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.822+0000 7f28a342e640 -1 librbd::ImageWatcher: 0x5610bd1b8c00 image watch failed: 94629889634304, (107) Transport endpoint is not connected 2023-10-20T09:46:01.822 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.822+0000 7f28a342e640 -1 librbd::Watcher: 0x5610bd1b8c00 handle_error: handle=94629889634304: (107) Transport endpoint is not connected 2023-10-20T09:46:01.828 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.826+0000 7f28a342e640 -1 librbd::watcher::RewatchRequest: 0x5610bbc0d7c0 handle_unwatch client blocklisted 2023-10-20T09:46:01.829 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.826+0000 7f28a342e640 -1 librbd::Watcher: 0x5610bd1b8c00 handle_rewatch: client blocklisted 2023-10-20T09:46:01.829 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.826+0000 7f28a342e640 -1 librbd::ManagedLock: 0x5610bc39a698 send_acquire_lock: watcher not registered - client blocklisted 2023-10-20T09:46:01.829 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.826+0000 7f28a342e640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5610bcd0a820 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:01.829 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.826+0000 7f28a342e640 -1 librbd::io::AioCompletion: 0x5610bd1bc160 fail: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:01.857 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.858+0000 7f28a3c2f640 -1 librbd::ImageWatcher: 0x5610bce91200 image watch failed: 94629890612224, (107) Transport endpoint is not connected 2023-10-20T09:46:01.857 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.858+0000 7f28a3c2f640 -1 librbd::Watcher: 0x5610bce91200 handle_error: handle=94629890612224: (107) Transport endpoint is not connected 2023-10-20T09:46:01.857 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.858+0000 7f28a3c2f640 -1 librbd::ImageWatcher: 0x5610bd1b8000 image watch failed: 94629890606080, (107) Transport endpoint is not connected 2023-10-20T09:46:01.858 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.858+0000 7f28a3c2f640 -1 librbd::Watcher: 0x5610bd1b8000 handle_error: handle=94629890606080: (107) Transport endpoint is not connected 2023-10-20T09:46:01.858 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.858+0000 7f28a3c2f640 -1 librbd::ImageWatcher: 0x5610bd1b8900 image watch failed: 94629892265984, (107) Transport endpoint is not connected 2023-10-20T09:46:01.858 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.858+0000 7f28a3c2f640 -1 librbd::Watcher: 0x5610bd1b8900 handle_error: handle=94629892265984: (107) Transport endpoint is not connected 2023-10-20T09:46:01.858 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.858+0000 7f28a3c2f640 -1 librbd::watcher::RewatchRequest: 0x5610bb98cbe0 handle_unwatch client blocklisted 2023-10-20T09:46:01.858 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.858+0000 7f28a3c2f640 -1 librbd::Watcher: 0x5610bce91200 handle_rewatch: client blocklisted 2023-10-20T09:46:01.859 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.858+0000 7f28a3c2f640 -1 librbd::ManagedLock: 0x5610bba10d18 send_acquire_lock: watcher not registered - client blocklisted 2023-10-20T09:46:01.859 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.858+0000 7f28a3c2f640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5610bcd0bc70 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:01.859 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.858+0000 7f28a3c2f640 -1 librbd::io::AioCompletion: 0x5610bd1f9600 fail: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:01.860 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.858+0000 7f28a342e640 -1 librbd::watcher::RewatchRequest: 0x5610bb98ea00 handle_unwatch client blocklisted 2023-10-20T09:46:01.860 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.858+0000 7f28a342e640 -1 librbd::Watcher: 0x5610bd1b8000 handle_rewatch: client blocklisted 2023-10-20T09:46:01.860 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.858+0000 7f28a342e640 -1 librbd::watcher::RewatchRequest: 0x5610bb98cc30 handle_unwatch client blocklisted 2023-10-20T09:46:01.860 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.858+0000 7f28a342e640 -1 librbd::Watcher: 0x5610bd1b8900 handle_rewatch: client blocklisted 2023-10-20T09:46:01.861 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.858+0000 7f28a342e640 -1 librbd::ManagedLock: 0x5610bc050698 send_acquire_lock: watcher not registered - client blocklisted 2023-10-20T09:46:01.861 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.858+0000 7f28a342e640 -1 librbd::ManagedLock: 0x5610bb877878 send_acquire_lock: watcher not registered - client blocklisted 2023-10-20T09:46:01.861 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.858+0000 7f28a342e640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5610bcd0a680 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:01.861 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.858+0000 7f28a342e640 -1 librbd::io::AioCompletion: 0x5610bd1efe40 fail: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:01.861 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.858+0000 7f28a342e640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5610bcd0b930 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:01.862 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.858+0000 7f28a342e640 -1 librbd::io::AioCompletion: 0x5610bd1f8c60 fail: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:01.862 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.858+0000 7f28a342e640 -1 librbd::ImageWatcher: 0x5610bd1b8600 image watch failed: 94629892267008, (107) Transport endpoint is not connected 2023-10-20T09:46:01.862 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.858+0000 7f28a342e640 -1 librbd::Watcher: 0x5610bd1b8600 handle_error: handle=94629892267008: (107) Transport endpoint is not connected 2023-10-20T09:46:01.862 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.858+0000 7f28a3c2f640 -1 librbd::watcher::RewatchRequest: 0x5610bbfdceb0 handle_unwatch client blocklisted 2023-10-20T09:46:01.862 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.858+0000 7f28a3c2f640 -1 librbd::Watcher: 0x5610bd1b8600 handle_rewatch: client blocklisted 2023-10-20T09:46:01.863 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.862+0000 7f28a3c2f640 -1 librbd::ImageWatcher: 0x5610bd1b9b00 image watch failed: 94629892307968, (107) Transport endpoint is not connected 2023-10-20T09:46:01.868 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.862+0000 7f28a3c2f640 -1 librbd::Watcher: 0x5610bd1b9b00 handle_error: handle=94629892307968: (107) Transport endpoint is not connected 2023-10-20T09:46:01.869 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.862+0000 7f28a342e640 -1 librbd::watcher::RewatchRequest: 0x5610bb98cd20 handle_unwatch client blocklisted 2023-10-20T09:46:01.869 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.862+0000 7f28a342e640 -1 librbd::Watcher: 0x5610bd1b9b00 handle_rewatch: client blocklisted 2023-10-20T09:46:01.870 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.862+0000 7f28a3c2f640 -1 librbd::ImageWatcher: 0x5610bce91500 image watch failed: 94629890610176, (107) Transport endpoint is not connected 2023-10-20T09:46:01.870 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.862+0000 7f28a3c2f640 -1 librbd::Watcher: 0x5610bce91500 handle_error: handle=94629890610176: (107) Transport endpoint is not connected 2023-10-20T09:46:01.870 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.862+0000 7f28a342e640 -1 librbd::watcher::RewatchRequest: 0x5610bb98ccd0 handle_unwatch client blocklisted 2023-10-20T09:46:01.870 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.862+0000 7f28a342e640 -1 librbd::Watcher: 0x5610bce91500 handle_rewatch: client blocklisted 2023-10-20T09:46:01.997 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.994+0000 7f28a3c2f640 -1 librbd::ImageWatcher: 0x5610bd0ee000 image watch failed: 94629892305920, (107) Transport endpoint is not connected 2023-10-20T09:46:01.998 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:01.998+0000 7f28a3c2f640 -1 librbd::Watcher: 0x5610bd0ee000 handle_error: handle=94629892305920: (107) Transport endpoint is not connected 2023-10-20T09:46:02.092 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.090+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve create_timestamp: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:02.092 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.090+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bce06880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:02.105 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.098+0000 7f28a3c2f640 -1 librbd::watcher::RewatchRequest: 0x5610bc576d20 handle_unwatch client blocklisted 2023-10-20T09:46:02.105 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.098+0000 7f28a3c2f640 -1 librbd::Watcher: 0x5610bd0ee000 handle_rewatch: client blocklisted 2023-10-20T09:46:02.105 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.098+0000 7f28a3c2f640 -1 librbd::ManagedLock: 0x5610bba21bb8 send_acquire_lock: watcher not registered - client blocklisted 2023-10-20T09:46:02.105 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.098+0000 7f28a3c2f640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5610bcd0af70 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:02.106 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.098+0000 7f28a3c2f640 -1 librbd::io::AioCompletion: 0x5610bcdd66e0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:02.809 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/389382636 until 2023-10-20T10:46:02.069346+0000 (3600 sec) 2023-10-20T09:46:02.821 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.822+0000 7f28a342e640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:02.821 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.822+0000 7f28a342e640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:02.821 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.822+0000 7f28a342e640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:02.821 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.822+0000 7f28a342e640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5610bcd0ba00 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:02.822 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.822+0000 7f28a342e640 -1 librbd::io::AioCompletion: 0x5610bd1bd1e0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:02.822 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.822+0000 7f28a342e640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5610bb8e1ba0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:02.822 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.822+0000 7f28a342e640 -1 librbd::io::AioCompletion: 0x5610bd1bd4a0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:02.822 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.822+0000 7f28a342e640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5610bcd0add0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:02.822 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.822+0000 7f28a342e640 -1 librbd::io::AioCompletion: 0x5610bd1f82c0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:02.845 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:46:02.859 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/389382636 2023-10-20T09:46:02.914 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.822+0000 7f28a342e640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:02.915 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.822+0000 7f28a342e640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:02.915 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.822+0000 7f28a342e640 -1 librbd::io::AioCompletion: 0x5610bd1bc160 fail: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:02.915 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.822+0000 7f28a342e640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5610bd1bea00 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:02.916 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.822+0000 7f28a342e640 -1 librbd::io::AioCompletion: 0x5610bd1f8c60 fail: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:02.916 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.822+0000 7f28a342e640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5610bbf1b2c0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:02.916 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.822+0000 7f28a342e640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:02.916 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.822+0000 7f28a342e640 -1 librbd::managed_lock::ReleaseRequest: 0x5610bb8876e0 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:02.916 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.822+0000 7f28a342e640 -1 librbd::managed_lock::ReleaseRequest: 0x5610bbac5c20 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:02.917 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.822+0000 7f28a342e640 -1 librbd::io::AioCompletion: 0x5610bd1f8f20 fail: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:02.917 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.822+0000 7f28a342e640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x5610bd1b00c0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:02.917 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:46:02.822+0000 7f28a3c2f640 -1 librbd::managed_lock::ReleaseRequest: 0x5610bb8876e0 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:46:03.528 INFO:tasks.workunit.client.0.smithi150.stderr:listed 27 entries 2023-10-20T09:46:03.564 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/389382636 2023-10-20T09:46:03.564 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:46:13.567 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:46:13.567 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:46:13.578 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:46:13.578 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:46:14.420 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/389382636 2023-10-20T09:46:14.421 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:46:14.421 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795174 2023-10-20T09:46:14.422 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:46:14.422 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/389382636 ]] 2023-10-20T09:46:14.423 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/389382636 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\8\9\3\8\2\6\3\6 ]] 2023-10-20T09:46:14.423 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:46:24.428 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:46:24.428 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:46:24.428 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:46:24.437 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:46:25.348 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/389382636 2023-10-20T09:46:25.348 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:46:25.349 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795185 2023-10-20T09:46:25.349 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:46:25.349 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/389382636 ]] 2023-10-20T09:46:25.350 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/389382636 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\8\9\3\8\2\6\3\6 ]] 2023-10-20T09:46:25.350 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:46:35.353 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:46:35.354 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:46:35.356 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:46:35.357 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:46:36.240 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/389382636 2023-10-20T09:46:36.241 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:46:36.241 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795196 2023-10-20T09:46:36.241 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:46:36.241 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/389382636 ]] 2023-10-20T09:46:36.242 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/389382636 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\8\9\3\8\2\6\3\6 ]] 2023-10-20T09:46:36.242 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:46:46.247 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:46:46.248 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:46:46.256 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:46:46.260 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:46:47.219 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/389382636 2023-10-20T09:46:47.219 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:46:47.220 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795207 2023-10-20T09:46:47.220 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:46:47.221 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/389382636 ]] 2023-10-20T09:46:47.221 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/389382636 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\8\9\3\8\2\6\3\6 ]] 2023-10-20T09:46:47.221 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:46:48.638 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:46:48.641+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:46:48.639 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:46:48.641+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T09:46:48.639 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:46:48.641+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:46:48.648 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:46:48.649+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T09:46:48.648 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:46:48.649+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:46:48.648 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:46:48.649+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:46:48.648 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:46:48.649+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:46:48.648 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:46:48.649+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:46:55.969 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:46:55.973+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:46:55.970 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:46:55.973+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T09:46:55.970 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:46:55.973+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:46:55.972 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:46:55.973+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T09:46:55.972 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:46:55.973+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:46:55.972 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:46:55.973+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:46:55.973 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:46:55.973+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:46:55.973 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:46:55.973+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:46:57.222 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:46:57.222 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:46:57.232 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:46:57.240 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:46:58.195 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/389382636 2023-10-20T09:46:58.196 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:46:58.197 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795218 2023-10-20T09:46:58.198 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:46:58.198 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/389382636 ]] 2023-10-20T09:46:58.198 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/389382636 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\8\9\3\8\2\6\3\6 ]] 2023-10-20T09:46:58.198 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:47:03.077 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:47:03.073+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:47:08.199 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:47:08.200 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:47:08.204 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:47:08.205 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:47:09.166 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/249834779 2023-10-20T09:47:09.168 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:47:09.169 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795229 2023-10-20T09:47:09.169 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:47:09.169 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/249834779 ]] 2023-10-20T09:47:09.169 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/249834779 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\8\9\3\8\2\6\3\6 ]] 2023-10-20T09:47:09.170 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/249834779 2023-10-20T09:47:11.350 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/249834779 until 2023-10-20T10:47:10.601511+0000 (3600 sec) 2023-10-20T09:47:11.384 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:47:11.385 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/249834779 2023-10-20T09:47:12.098 INFO:tasks.workunit.client.0.smithi150.stderr:listed 28 entries 2023-10-20T09:47:12.148 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/249834779 2023-10-20T09:47:12.148 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:47:14.962 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:47:14.960+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:47:14.969 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:47:14.968+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:47:14.973 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:47:14.976+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T09:47:14.973 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:47:14.976+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:47:14.973 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:47:14.976+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:47:22.153 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:47:22.154 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:47:22.158 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:47:22.163 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:47:23.064 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/249834779 2023-10-20T09:47:23.065 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:47:23.066 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795243 2023-10-20T09:47:23.066 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:47:23.067 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/249834779 ]] 2023-10-20T09:47:23.067 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/249834779 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\4\9\8\3\4\7\7\9 ]] 2023-10-20T09:47:23.067 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:47:33.067 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:47:33.067 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:47:33.068 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:47:33.068 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:47:33.851 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/249834779 2023-10-20T09:47:33.852 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:47:33.852 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795253 2023-10-20T09:47:33.853 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:47:33.853 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/249834779 ]] 2023-10-20T09:47:33.853 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/249834779 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\4\9\8\3\4\7\7\9 ]] 2023-10-20T09:47:33.853 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:47:43.864 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:47:43.865 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:47:43.867 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:47:43.877 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:47:44.946 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/249834779 2023-10-20T09:47:44.947 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:47:44.948 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795264 2023-10-20T09:47:44.948 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:47:44.948 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/249834779 ]] 2023-10-20T09:47:44.949 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/249834779 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\4\9\8\3\4\7\7\9 ]] 2023-10-20T09:47:44.949 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:47:54.950 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:47:54.954 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:47:54.963 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:47:54.964 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:47:56.035 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/249834779 2023-10-20T09:47:56.036 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:47:56.036 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795276 2023-10-20T09:47:56.036 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:47:56.037 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/249834779 ]] 2023-10-20T09:47:56.037 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/249834779 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\4\9\8\3\4\7\7\9 ]] 2023-10-20T09:47:56.037 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:48:00.002 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.003+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.003 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.003+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc3b3780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.005 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.007+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.005 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.007+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc919a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.011 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.011+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.012 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.011+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bb8f0080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.017 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.019+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.017 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.019+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc918400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.021 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.023+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.022 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.023+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcf5c600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.027 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.027+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.027 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.027+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcf5da00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.047 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.047+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.047 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.047+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcf5dd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.054 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.055+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.055 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.055+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcf5ce80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.066 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.067+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.067 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.067+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcf5de00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.079 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.079+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.079 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.079+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcf5db80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.085 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.087+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.085 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.087+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcf5d800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.091 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.091+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.091 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.091+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcf5ce00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.109 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.111+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.110 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.111+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc788100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.116 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.115+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.116 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.115+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc788300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.122 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.123+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.123 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.123+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc788500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.128 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.127+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.129 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.131+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc788700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.212 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.135+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.212 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.135+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc788900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.214 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.211+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.214 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.211+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.214 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.211+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.214 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.211+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc788b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.215 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.211+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc788d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:00.215 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:48:00.211+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc788f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:48:06.052 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:48:06.062 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:48:06.069 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:48:06.070 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:48:06.937 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/249834779 2023-10-20T09:48:06.937 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:48:06.938 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795286 2023-10-20T09:48:06.938 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:48:06.939 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/249834779 ]] 2023-10-20T09:48:06.939 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/249834779 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\4\9\8\3\4\7\7\9 ]] 2023-10-20T09:48:06.939 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:48:13.342 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:48:13.343+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:48:13.342 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:48:13.343+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T09:48:13.343 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:48:13.343+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:48:13.343 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:48:13.343+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab4f70 2023-10-20T09:48:13.343 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:48:13.343+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T09:48:13.343 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:48:13.343+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:48:13.343 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:48:13.343+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:48:13.344 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:48:13.343+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T09:48:13.344 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:48:13.343+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:48:13.344 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:48:13.343+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:48:16.947 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:48:16.949 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:48:16.956 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:48:16.960 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:48:17.933 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/249834779 2023-10-20T09:48:17.933 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:48:17.934 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795297 2023-10-20T09:48:17.934 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:48:17.934 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/249834779 ]] 2023-10-20T09:48:17.935 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/249834779 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\4\9\8\3\4\7\7\9 ]] 2023-10-20T09:48:17.935 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:48:27.942 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:48:27.942 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:48:27.948 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:48:27.956 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:48:28.820 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/171431162 2023-10-20T09:48:28.821 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:48:28.822 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795308 2023-10-20T09:48:28.822 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:48:28.822 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/171431162 ]] 2023-10-20T09:48:28.823 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/171431162 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\2\4\9\8\3\4\7\7\9 ]] 2023-10-20T09:48:28.823 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/171431162 2023-10-20T09:48:30.841 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/171431162 until 2023-10-20T10:48:30.095275+0000 (3600 sec) 2023-10-20T09:48:30.869 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/171431162 2023-10-20T09:48:30.871 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:48:31.690 INFO:tasks.workunit.client.0.smithi150.stderr:listed 29 entries 2023-10-20T09:48:31.721 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/171431162 2023-10-20T09:48:31.721 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:48:41.730 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:48:41.730 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:48:41.730 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:48:41.732 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:48:42.614 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/171431162 2023-10-20T09:48:42.621 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:48:42.622 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795322 2023-10-20T09:48:42.622 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:48:42.623 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/171431162 ]] 2023-10-20T09:48:42.623 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/171431162 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\7\1\4\3\1\1\6\2 ]] 2023-10-20T09:48:42.623 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:48:52.624 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:48:52.625 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:48:52.625 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:48:52.632 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:48:53.626 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/171431162 2023-10-20T09:48:53.626 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:48:53.627 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795333 2023-10-20T09:48:53.628 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:48:53.628 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/171431162 ]] 2023-10-20T09:48:53.628 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/171431162 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\7\1\4\3\1\1\6\2 ]] 2023-10-20T09:48:53.629 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:49:00.001 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.002+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.001 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.002+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bce07e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.004 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.006+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.005 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.006+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc920e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.038 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.038+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.038 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.038+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcc47180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.078 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.078+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.078 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.078+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcb32e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.079 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.078+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.079 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.078+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcc54d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.084 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.082+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.084 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.082+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bbbd6080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.089 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.090+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.089 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.090+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcf5dd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.093 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.094+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.093 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.094+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcf5d480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.099 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.098+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.099 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.098+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc788a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.104 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.106+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.105 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.106+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc788f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.111 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.110+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.111 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.110+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcf5de80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.113 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.114+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.113 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.114+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcf5cd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.126 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.126+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.126 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.126+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.126 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.126+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc789280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.127 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.126+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc789080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.130 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.130+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.131 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.130+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc789680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.136 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.134+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.136 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.134+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc789880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.142 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.142+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.142 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.142+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc789a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.148 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.146+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.148 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.146+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc789c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.154 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.154+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.154 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.154+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc789e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.160 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.158+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:00.160 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:00.158+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bd080080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:49:03.628 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:49:03.629 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:49:03.631 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:49:03.633 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:49:04.707 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/171431162 2023-10-20T09:49:04.720 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:49:04.722 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795344 2023-10-20T09:49:04.722 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:49:04.722 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/171431162 ]] 2023-10-20T09:49:04.722 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/171431162 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\7\1\4\3\1\1\6\2 ]] 2023-10-20T09:49:04.722 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:49:14.738 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:49:14.739 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:49:14.739 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:49:14.747 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:49:15.592 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/171431162 2023-10-20T09:49:15.595 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:49:15.596 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795355 2023-10-20T09:49:15.596 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:49:15.596 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/171431162 ]] 2023-10-20T09:49:15.597 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/171431162 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\7\1\4\3\1\1\6\2 ]] 2023-10-20T09:49:15.597 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:49:25.598 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:49:25.598 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:49:25.603 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:49:25.617 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:49:26.454 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1658214644 2023-10-20T09:49:26.454 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:49:26.455 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795366 2023-10-20T09:49:26.455 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:49:26.455 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1658214644 ]] 2023-10-20T09:49:26.455 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1658214644 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\7\1\4\3\1\1\6\2 ]] 2023-10-20T09:49:26.456 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/1658214644 2023-10-20T09:49:28.407 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/1658214644 until 2023-10-20T10:49:27.697066+0000 (3600 sec) 2023-10-20T09:49:28.483 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:49:28.484 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/1658214644 2023-10-20T09:49:29.285 INFO:tasks.workunit.client.0.smithi150.stderr:listed 30 entries 2023-10-20T09:49:29.335 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/1658214644 2023-10-20T09:49:29.336 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:49:39.337 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:49:39.339 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:49:39.339 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:49:39.348 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:49:40.268 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1658214644 2023-10-20T09:49:40.273 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:49:40.274 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795380 2023-10-20T09:49:40.275 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:49:40.275 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1658214644 ]] 2023-10-20T09:49:40.275 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1658214644 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\6\5\8\2\1\4\6\4\4 ]] 2023-10-20T09:49:40.275 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:49:50.277 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:49:50.277 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:49:50.277 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:49:50.278 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:49:51.213 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1658214644 2023-10-20T09:49:51.214 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:49:51.214 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795391 2023-10-20T09:49:51.215 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:49:51.215 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1658214644 ]] 2023-10-20T09:49:51.215 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1658214644 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\6\5\8\2\1\4\6\4\4 ]] 2023-10-20T09:49:51.215 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:50:00.001 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:59.997+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.001 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:49:59.997+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc54b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.004 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.001+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.004 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.001+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcf5dd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.015 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.013+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.016 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.013+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc788d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.018 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.017+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.019 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.017+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc788b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.034 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.033+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.034 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.033+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd080700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.043 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.041+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.044 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.041+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd080580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.060 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.057+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.060 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.057+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd080a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.066 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.065+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.066 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.065+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd080880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.096 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.093+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.096 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.093+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd080f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.105 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.105+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.105 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.105+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bd080e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.114 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.113+0000 7f28a2c2d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.114 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.113+0000 7f28a2c2d640 -1 librbd::ImageState: 0x5610bd080c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.121 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.121+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.121 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.121+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd080300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.133 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.133+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.133 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.133+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd081200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.135 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.133+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.135 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.133+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd081400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.138 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.137+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.138 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.137+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd081600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.141 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.141+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.141 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.141+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd081800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.144 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.141+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.144 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.141+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd081a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.146 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.145+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.146 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.145+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd081c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.149 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.149+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.150 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.149+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd081e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.152 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.149+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:00.153 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:50:00.153+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc78000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:50:01.219 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:50:01.221 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:50:01.239 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:50:01.242 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:50:02.131 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1658214644 2023-10-20T09:50:02.132 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:50:02.132 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795402 2023-10-20T09:50:02.133 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:50:02.133 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1658214644 ]] 2023-10-20T09:50:02.133 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1658214644 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\6\5\8\2\1\4\6\4\4 ]] 2023-10-20T09:50:02.133 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:50:11.169 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:50:11.161+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:50:11.182 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:50:11.177+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T09:50:11.182 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:50:11.177+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:50:11.183 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:50:11.177+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:50:12.133 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:50:12.134 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:50:12.134 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:50:12.135 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:50:13.056 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1658214644 2023-10-20T09:50:13.056 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:50:13.057 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795413 2023-10-20T09:50:13.057 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:50:13.057 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1658214644 ]] 2023-10-20T09:50:13.057 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1658214644 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\6\5\8\2\1\4\6\4\4 ]] 2023-10-20T09:50:13.058 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:50:18.814 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:50:18.813+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:50:18.815 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:50:18.813+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:50:23.058 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:50:23.059 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:50:23.072 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:50:23.078 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:50:23.976 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1658214644 2023-10-20T09:50:23.976 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:50:23.977 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795423 2023-10-20T09:50:23.977 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:50:23.977 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1658214644 ]] 2023-10-20T09:50:23.977 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1658214644 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\6\5\8\2\1\4\6\4\4 ]] 2023-10-20T09:50:23.977 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:50:33.982 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:50:33.994 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:50:34.000 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:50:34.006 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:50:35.188 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/1658214644 2023-10-20T09:50:35.197 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:50:35.198 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795435 2023-10-20T09:50:35.198 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:50:35.199 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/1658214644 ]] 2023-10-20T09:50:35.199 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/1658214644 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\6\5\8\2\1\4\6\4\4 ]] 2023-10-20T09:50:35.199 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:50:45.206 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:50:45.207 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:50:45.208 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:50:45.214 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:50:46.125 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3966202163 2023-10-20T09:50:46.126 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:50:46.126 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795446 2023-10-20T09:50:46.127 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:50:46.127 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3966202163 ]] 2023-10-20T09:50:46.127 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3966202163 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\1\6\5\8\2\1\4\6\4\4 ]] 2023-10-20T09:50:46.127 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/3966202163 2023-10-20T09:50:47.848 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/3966202163 until 2023-10-20T10:50:47.116605+0000 (3600 sec) 2023-10-20T09:50:47.910 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:50:47.912 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/3966202163 2023-10-20T09:50:48.758 INFO:tasks.workunit.client.0.smithi150.stderr:listed 31 entries 2023-10-20T09:50:48.791 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/3966202163 2023-10-20T09:50:48.791 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:50:52.396 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:50:52.396+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:50:52.396 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:50:52.396+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:50:58.795 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:50:58.796 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:50:58.800 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:50:58.801 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:50:59.656 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3966202163 2023-10-20T09:50:59.656 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:50:59.657 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795459 2023-10-20T09:50:59.657 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:50:59.657 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3966202163 ]] 2023-10-20T09:50:59.658 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3966202163 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\9\6\6\2\0\2\1\6\3 ]] 2023-10-20T09:50:59.658 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:51:00.000 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.000+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.001 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.000+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bcdf7880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.014 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.016+0000 7f28a3c2f640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.015 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.016+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.015 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.016+0000 7f28a3c2f640 -1 librbd::ImageState: 0x5610bc211100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.015 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.016+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc55a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.020 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.020+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.020 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.020+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc210800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.037 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.040+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.037 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.040+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcada280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.050 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.052+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.051 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.052+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bbf9d500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.054 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.056+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.054 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.056+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcf5df00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.057 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.060+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.057 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.060+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcf5c500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.068 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.068+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.068 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.068+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc788a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.085 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.088+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.085 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.088+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc788800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.086 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.088+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.086 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.088+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bce07e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.093 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.096+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.093 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.096+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcf5d800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.100 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.100+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.100 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.100+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd081300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.103 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.104+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.103 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.104+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bbbd7d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.119 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.120+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.119 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.120+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc788d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.131 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.132+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.131 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.132+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd081500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.133 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.136+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.133 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.136+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd081700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.135 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.136+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.136 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.136+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd081900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.138 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.140+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.138 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.140+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd081f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.140 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.140+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:00.140 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:51:00.140+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd081d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:51:05.055 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:51:05.052+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:51:05.055 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:51:05.052+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:51:05.056 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:51:05.056+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:51:09.665 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:51:09.665 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:51:09.666 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:51:09.666 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:51:10.654 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3966202163 2023-10-20T09:51:10.656 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:51:10.657 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795470 2023-10-20T09:51:10.657 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:51:10.658 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3966202163 ]] 2023-10-20T09:51:10.658 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3966202163 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\9\6\6\2\0\2\1\6\3 ]] 2023-10-20T09:51:10.658 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:51:20.659 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:51:20.659 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:51:20.667 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:51:20.671 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:51:21.509 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3966202163 2023-10-20T09:51:21.509 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:51:21.510 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795481 2023-10-20T09:51:21.510 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:51:21.510 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3966202163 ]] 2023-10-20T09:51:21.511 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3966202163 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\9\6\6\2\0\2\1\6\3 ]] 2023-10-20T09:51:21.511 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:51:31.513 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:51:31.513 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:51:31.518 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:51:31.524 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:51:32.469 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3966202163 2023-10-20T09:51:32.470 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:51:32.471 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795492 2023-10-20T09:51:32.471 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:51:32.471 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3966202163 ]] 2023-10-20T09:51:32.472 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3966202163 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\9\6\6\2\0\2\1\6\3 ]] 2023-10-20T09:51:32.472 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:51:42.473 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:51:42.473 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:51:42.475 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:51:42.487 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:51:43.434 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3966202163 2023-10-20T09:51:43.439 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:51:43.447 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795503 2023-10-20T09:51:43.447 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:51:43.447 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3966202163 ]] 2023-10-20T09:51:43.447 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3966202163 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\9\6\6\2\0\2\1\6\3 ]] 2023-10-20T09:51:43.448 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:51:53.448 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:51:53.449 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:51:53.449 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:51:53.450 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:51:54.436 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/4059769879 2023-10-20T09:51:54.436 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:51:54.437 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795514 2023-10-20T09:51:54.437 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:51:54.437 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/4059769879 ]] 2023-10-20T09:51:54.437 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/4059769879 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\9\6\6\2\0\2\1\6\3 ]] 2023-10-20T09:51:54.438 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/4059769879 2023-10-20T09:51:56.431 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/4059769879 until 2023-10-20T10:51:55.653270+0000 (3600 sec) 2023-10-20T09:51:56.465 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:51:56.465 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/4059769879 2023-10-20T09:51:57.124 INFO:tasks.workunit.client.0.smithi150.stderr:listed 32 entries 2023-10-20T09:51:57.147 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/4059769879 2023-10-20T09:51:57.147 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:52:00.001 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.003+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.001 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.003+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bb8d3980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.004 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.007+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.004 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.007+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcdf7c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.007 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.007+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.007 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.007+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc918280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.010 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.011+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.010 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.011+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bbbd6a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.016 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.019+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.017 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.019+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc919680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.024 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.023+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.024 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.023+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcc54b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.033 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.035+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.034 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.035+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc789980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.045 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.047+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.045 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.047+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc789c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.051 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.051+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.051 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.051+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc918600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.056 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.059+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.057 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.059+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcf5d880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.062 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.063+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.062 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.063+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bcf5cf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.068 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.071+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.068 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.071+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bc789a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.075 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.075+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.075 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.075+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd081e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.081 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.083+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.081 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.083+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd081080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.086 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.087+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.087 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.087+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd0f4080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.093 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.095+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.094 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.095+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd0f4280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.099 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.099+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.100 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.099+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd0f4480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.124 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.123+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.124 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.123+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd0f4e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.134 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.135+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.135 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.135+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd0f4d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.155 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.155+0000 7f28a342e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:00.156 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T09:52:00.155+0000 7f28a342e640 -1 librbd::ImageState: 0x5610bd0f4b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-20T09:52:07.149 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:52:07.149 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:52:07.152 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:52:07.153 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:52:08.175 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/4059769879 2023-10-20T09:52:08.175 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:52:08.176 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795528 2023-10-20T09:52:08.176 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:52:08.176 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/4059769879 ]] 2023-10-20T09:52:08.177 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/4059769879 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\4\0\5\9\7\6\9\8\7\9 ]] 2023-10-20T09:52:08.177 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:52:16.376 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:52:16.375+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T09:52:16.377 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:52:16.375+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:52:16.377 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:52:16.375+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:52:16.377 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:52:16.375+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:52:16.377 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:52:16.375+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:52:18.184 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:52:18.198 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:52:18.209 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:52:18.209 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:52:19.035 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/4059769879 2023-10-20T09:52:19.036 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:52:19.037 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795539 2023-10-20T09:52:19.037 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:52:19.037 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/4059769879 ]] 2023-10-20T09:52:19.037 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/4059769879 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\4\0\5\9\7\6\9\8\7\9 ]] 2023-10-20T09:52:19.037 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:52:29.038 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:52:29.039 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:52:29.053 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:52:29.068 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:52:29.996 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/4059769879 2023-10-20T09:52:29.996 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:52:29.997 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795550 2023-10-20T09:52:29.997 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:52:29.997 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/4059769879 ]] 2023-10-20T09:52:29.997 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/4059769879 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\4\0\5\9\7\6\9\8\7\9 ]] 2023-10-20T09:52:29.998 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:52:39.999 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:52:39.999 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:52:39.999 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:52:40.018 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:52:41.010 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/4059769879 2023-10-20T09:52:41.010 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:52:41.012 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795561 2023-10-20T09:52:41.012 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:52:41.012 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/4059769879 ]] 2023-10-20T09:52:41.013 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/4059769879 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\4\0\5\9\7\6\9\8\7\9 ]] 2023-10-20T09:52:41.013 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:52:51.012 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:52:51.012 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:52:51.013 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:52:51.034 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:52:51.918 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/4059769879 2023-10-20T09:52:51.918 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:52:51.919 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795571 2023-10-20T09:52:51.919 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:52:51.919 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/4059769879 ]] 2023-10-20T09:52:51.920 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/4059769879 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\4\0\5\9\7\6\9\8\7\9 ]] 2023-10-20T09:52:51.920 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:53:01.008 INFO:tasks.ceph.osd.2.smithi150.stderr:2023-10-20T09:53:01.006+0000 7f12576b6640 -1 reset not still connected to 0x5635c0e90410 2023-10-20T09:53:01.930 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:53:01.934 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:53:01.934 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:53:01.939 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:53:02.804 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:53:02.805 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:53:02.805 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795582 2023-10-20T09:53:02.805 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:53:02.806 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:53:02.806 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\4\0\5\9\7\6\9\8\7\9 ]] 2023-10-20T09:53:02.806 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist add 172.21.15.150:0/3774459303 2023-10-20T09:53:04.446 INFO:tasks.workunit.client.0.smithi150.stderr:blocklisting 172.21.15.150:0/3774459303 until 2023-10-20T10:53:03.640170+0000 (3600 sec) 2023-10-20T09:53:04.492 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd blocklist ls 2023-10-20T09:53:04.495 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 172.21.15.150:0/3774459303 2023-10-20T09:53:05.228 INFO:tasks.workunit.client.0.smithi150.stderr:listed 33 entries 2023-10-20T09:53:05.268 INFO:tasks.workunit.client.0.smithi150.stderr:+ PREV_CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:53:05.269 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:53:15.271 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:53:15.271 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:53:15.272 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:53:15.277 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:53:16.194 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:53:16.195 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:53:16.196 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795596 2023-10-20T09:53:16.196 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:53:16.196 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:53:16.196 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:53:16.196 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:53:21.452 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:53:21.450+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:53:22.645 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:53:22.646+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:53:26.198 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:53:26.198 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:53:26.206 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:53:26.209 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:53:27.152 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:53:27.152 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:53:27.154 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795607 2023-10-20T09:53:27.154 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:53:27.154 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:53:27.154 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:53:27.155 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:53:37.157 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:53:37.157 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:53:37.157 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:53:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:53:37.520 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:53:37.513+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T09:53:37.520 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:53:37.513+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:53:37.526 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:53:37.525+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T09:53:37.526 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:53:37.525+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:53:37.526 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:53:37.525+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:53:37.536 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:53:37.533+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:53:37.536 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:53:37.533+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:53:38.101 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:53:38.102 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:53:38.104 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795618 2023-10-20T09:53:38.104 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:53:38.104 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:53:38.104 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:53:38.104 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:53:39.253 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:53:39.253+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:53:39.253 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:53:39.253+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T09:53:39.253 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:53:39.253+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:53:39.257 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:53:39.257+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T09:53:39.257 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:53:39.257+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:53:39.257 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:53:39.257+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:53:39.258 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:53:39.257+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T09:53:39.258 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:53:39.257+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:53:39.258 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:53:39.257+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:53:48.107 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:53:48.107 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:53:48.110 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:53:48.116 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:53:49.062 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:53:49.063 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:53:49.064 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795629 2023-10-20T09:53:49.064 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:53:49.064 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:53:49.064 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:53:49.064 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:53:50.587 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:53:50.581+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:53:50.591 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:53:50.589+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:53:50.600 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:53:50.601+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:53:52.609 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:53:52.609+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:53:52.609 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:53:52.609+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:53:52.609 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:53:52.609+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:53:52.609 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:53:52.609+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:53:59.066 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:53:59.066 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:53:59.067 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:53:59.067 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:53:59.856 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:53:59.857 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:53:59.859 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795639 2023-10-20T09:53:59.859 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:53:59.859 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:53:59.859 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:53:59.859 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:54:03.654 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:54:03.653+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:54:03.654 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:54:03.653+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:54:03.661 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:54:03.657+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:54:09.861 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:54:09.862 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:54:09.870 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:54:09.878 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:54:10.761 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:54:10.762 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:54:10.762 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795650 2023-10-20T09:54:10.762 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:54:10.762 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:54:10.763 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:54:10.763 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:54:20.765 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:54:20.770 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:54:20.778 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:54:20.778 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:54:21.645 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:54:21.646 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:54:21.647 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795661 2023-10-20T09:54:21.647 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:54:21.648 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:54:21.648 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:54:21.648 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:54:23.507 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:54:23.509+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:54:23.525 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:54:23.529+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:54:23.525 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:54:23.529+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:54:31.649 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:54:31.650 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:54:31.663 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:54:31.664 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:54:32.637 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:54:32.637 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:54:32.638 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795672 2023-10-20T09:54:32.638 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:54:32.639 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:54:32.639 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:54:32.639 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:54:42.654 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:54:42.655 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:54:42.655 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:54:42.656 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:54:43.532 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:54:43.533 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:54:43.534 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795683 2023-10-20T09:54:43.535 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:54:43.535 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:54:43.535 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:54:43.536 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:54:53.536 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:54:53.537 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:54:53.537 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:54:53.542 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:54:54.377 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:54:54.378 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:54:54.378 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795694 2023-10-20T09:54:54.379 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:54:54.379 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:54:54.379 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:54:54.379 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:55:04.393 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:55:04.394 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:55:04.394 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:55:04.399 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:55:05.322 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:55:05.328 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:55:05.329 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795705 2023-10-20T09:55:05.329 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:55:05.329 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:55:05.330 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:55:05.330 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:55:15.337 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:55:15.339 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:55:15.342 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:55:15.357 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:55:16.232 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:55:16.232 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:55:16.233 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795716 2023-10-20T09:55:16.233 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:55:16.233 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:55:16.233 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:55:16.234 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:55:26.239 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:55:26.244 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:55:26.248 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:55:26.251 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:55:27.093 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:55:27.093 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:55:27.094 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795727 2023-10-20T09:55:27.094 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:55:27.094 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:55:27.094 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:55:27.094 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:55:33.260 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:55:33.260+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:55:33.260 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:55:33.260+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T09:55:33.260 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:55:33.260+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:55:33.267 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:55:33.268+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T09:55:33.267 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:55:33.268+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:55:33.268 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:55:33.268+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:55:33.268 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:55:33.268+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T09:55:33.268 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:55:33.268+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:55:33.268 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:55:33.268+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:55:37.099 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:55:37.100 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:55:37.110 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:55:37.114 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:55:38.002 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:55:38.003 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:55:38.003 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795738 2023-10-20T09:55:38.004 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:55:38.004 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:55:38.004 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:55:38.004 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:55:48.005 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:55:48.006 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:55:48.007 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:55:48.025 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:55:49.014 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:55:49.014 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:55:49.015 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795749 2023-10-20T09:55:49.015 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:55:49.015 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:55:49.016 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:55:49.016 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:55:59.016 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:55:59.017 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:55:59.017 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:55:59.019 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:56:00.080 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:56:00.084 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:56:00.086 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795760 2023-10-20T09:56:00.086 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:56:00.086 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:56:00.086 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:56:00.087 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:56:07.455 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:56:07.443+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T09:56:07.459 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:56:07.447+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:56:07.461 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:56:07.447+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:56:10.094 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:56:10.099 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:56:10.107 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:56:10.108 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:56:10.946 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:56:10.946 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:56:10.947 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795770 2023-10-20T09:56:10.947 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:56:10.947 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:56:10.947 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:56:10.947 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:56:12.085 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:56:12.087+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:56:12.085 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:56:12.087+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T09:56:20.949 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:56:20.949 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:56:20.970 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:56:20.979 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:56:21.840 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:56:21.841 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:56:21.842 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795781 2023-10-20T09:56:21.842 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:56:21.842 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:56:21.842 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:56:21.842 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:56:31.844 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:56:31.844 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:56:31.844 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:56:31.847 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:56:32.816 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:56:32.816 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:56:32.817 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795792 2023-10-20T09:56:32.817 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:56:32.817 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:56:32.817 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:56:32.818 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:56:42.818 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:56:42.819 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:56:42.819 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:56:42.824 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:56:43.663 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:56:43.664 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:56:43.664 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795803 2023-10-20T09:56:43.664 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:56:43.664 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:56:43.665 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:56:43.665 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:56:53.666 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:56:53.666 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:56:53.669 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:56:53.683 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:56:54.582 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:56:54.584 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:56:54.585 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795814 2023-10-20T09:56:54.585 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:56:54.585 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:56:54.586 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:56:54.586 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:57:02.880 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:57:02.878+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:57:02.886 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:57:02.886+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:57:04.588 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:57:04.592 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:57:04.595 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:57:04.595 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:57:05.453 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:57:05.453 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:57:05.454 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795825 2023-10-20T09:57:05.454 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:57:05.454 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:57:05.455 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:57:05.455 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:57:12.339 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:57:12.334+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:57:12.358 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:57:12.354+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:57:15.455 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:57:15.460 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:57:15.461 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:57:15.471 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:57:16.475 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:57:16.476 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:57:16.477 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795836 2023-10-20T09:57:16.477 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:57:16.477 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:57:16.477 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:57:16.478 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:57:26.478 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:57:26.478 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:57:26.479 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:57:26.479 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:57:27.392 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:57:27.393 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:57:27.393 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795847 2023-10-20T09:57:27.393 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:57:27.394 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:57:27.394 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:57:27.394 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:57:35.989 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:57:35.990+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:57:35.998 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:57:35.998+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T09:57:35.998 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:57:35.998+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:57:37.401 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:57:37.401 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:57:37.407 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:57:37.411 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:57:38.309 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:57:38.309 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:57:38.310 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795858 2023-10-20T09:57:38.310 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:57:38.310 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:57:38.310 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:57:38.311 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:57:46.059 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:57:46.053+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:57:46.059 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:57:46.057+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:57:46.069 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:57:46.069+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:57:46.069 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:57:46.069+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T09:57:48.323 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:57:48.324 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:57:48.343 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:57:48.344 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:57:49.324 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:57:49.324 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:57:49.325 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795869 2023-10-20T09:57:49.326 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:57:49.326 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:57:49.326 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:57:49.326 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:57:50.375 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:57:50.369+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:57:50.375 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:57:50.373+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:57:50.385 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:57:50.385+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:57:59.327 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:57:59.328 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:57:59.329 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:57:59.331 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:58:00.268 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:58:00.268 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:58:00.269 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795880 2023-10-20T09:58:00.269 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:58:00.270 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:58:00.270 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:58:00.270 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:58:10.281 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:58:10.283 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:58:10.288 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:58:10.299 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:58:11.210 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:58:11.210 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:58:11.211 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795891 2023-10-20T09:58:11.211 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:58:11.211 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:58:11.212 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:58:11.212 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:58:21.213 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:58:21.213 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:58:21.224 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:58:21.224 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:58:22.038 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:58:22.047 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:58:22.048 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795902 2023-10-20T09:58:22.048 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:58:22.049 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:58:22.049 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:58:22.049 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:58:32.065 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:58:32.071 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:58:32.071 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:58:32.073 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:58:33.074 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:58:33.077 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:58:33.079 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795913 2023-10-20T09:58:33.079 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:58:33.079 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:58:33.080 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:58:33.080 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:58:43.080 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:58:43.080 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:58:43.083 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:58:43.091 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:58:43.879 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:58:43.879 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:58:43.880 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795923 2023-10-20T09:58:43.880 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:58:43.880 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:58:43.880 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:58:43.880 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:58:53.889 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:58:53.889 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:58:53.890 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:58:53.891 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:58:54.820 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:58:54.820 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:58:54.821 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795934 2023-10-20T09:58:54.821 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:58:54.821 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:58:54.821 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:58:54.821 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:59:04.826 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:59:04.831 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:59:04.840 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:59:04.848 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:59:05.812 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:59:05.812 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:59:05.816 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795945 2023-10-20T09:59:05.816 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:59:05.817 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:59:05.817 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:59:05.817 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:59:15.819 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:59:15.819 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:59:15.824 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:59:15.835 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:59:16.850 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:59:16.850 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:59:16.851 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795956 2023-10-20T09:59:16.852 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:59:16.852 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:59:16.852 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:59:16.852 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:59:26.853 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:59:26.856 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:59:26.859 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:59:26.874 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:59:27.726 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:59:27.726 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:59:27.726 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795967 2023-10-20T09:59:27.727 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:59:27.727 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:59:27.727 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:59:27.727 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:59:37.746 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:59:37.755 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:59:37.768 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:59:37.768 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:59:38.742 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:59:38.742 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:59:38.742 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795978 2023-10-20T09:59:38.743 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:59:38.743 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:59:38.743 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:59:38.743 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:59:41.594 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:59:41.588+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T09:59:41.599 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:59:41.604+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T09:59:41.606 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T09:59:41.608+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T09:59:48.744 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:59:48.747 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T09:59:48.747 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:59:48.756 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:59:49.597 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T09:59:49.598 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T09:59:49.598 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697795989 2023-10-20T09:59:49.599 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T09:59:49.599 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T09:59:49.599 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T09:59:49.599 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T09:59:59.599 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T09:59:59.600 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T09:59:59.602 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T09:59:59.609 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:00:00.552 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:00:00.552 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:00:00.553 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796000 2023-10-20T10:00:00.553 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:00:00.553 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:00:00.553 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:00:00.553 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:00:04.328 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:00:04.331+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:00:04.328 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:00:04.331+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:00:10.555 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:00:10.555 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:00:10.564 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:00:10.570 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:00:11.479 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:00:11.479 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:00:11.480 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796011 2023-10-20T10:00:11.480 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:00:11.480 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:00:11.481 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:00:11.481 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:00:13.585 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:00:13.587+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:00:13.603 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:00:13.603+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T10:00:13.603 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:00:13.603+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:00:13.613 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:00:13.615+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab4f70 2023-10-20T10:00:13.614 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:00:13.615+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T10:00:13.614 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:00:13.615+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:00:13.615 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:00:13.615+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T10:00:13.615 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:00:13.615+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T10:00:13.615 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:00:13.615+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:00:13.615 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:00:13.615+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:00:21.481 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:00:21.482 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:00:21.483 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:00:21.490 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:00:22.325 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:00:22.327 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:00:22.328 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796022 2023-10-20T10:00:22.328 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:00:22.329 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:00:22.329 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:00:22.329 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:00:32.330 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:00:32.331 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:00:32.332 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:00:32.335 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:00:33.276 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:00:33.278 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:00:33.279 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796033 2023-10-20T10:00:33.280 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:00:33.280 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:00:33.280 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:00:33.280 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:00:43.281 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:00:43.282 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:00:43.283 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:00:43.295 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:00:44.271 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:00:44.279 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:00:44.280 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796044 2023-10-20T10:00:44.281 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:00:44.281 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:00:44.281 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:00:44.281 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:00:54.288 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:00:54.289 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:00:54.291 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:00:54.311 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:00:55.286 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:00:55.286 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:00:55.287 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796055 2023-10-20T10:00:55.287 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:00:55.287 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:00:55.287 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:00:55.287 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:01:05.288 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:01:05.289 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:01:05.290 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:01:05.301 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:01:06.299 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:01:06.300 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:01:06.300 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796066 2023-10-20T10:01:06.301 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:01:06.301 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:01:06.301 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:01:06.301 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:01:16.305 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:01:16.305 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:01:16.305 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:01:16.320 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:01:17.292 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:01:17.293 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:01:17.293 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796077 2023-10-20T10:01:17.293 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:01:17.294 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:01:17.294 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:01:17.294 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:01:23.912 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:01:23.913+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:01:23.913 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:01:23.913+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:01:27.295 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:01:27.295 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:01:27.303 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:01:27.307 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:01:28.386 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:01:28.387 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:01:28.389 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796088 2023-10-20T10:01:28.389 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:01:28.389 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:01:28.389 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:01:28.389 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:01:37.760 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:01:37.765+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:01:37.760 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:01:37.765+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:01:38.396 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:01:38.396 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:01:38.398 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:01:38.402 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:01:39.244 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:01:39.244 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:01:39.245 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796099 2023-10-20T10:01:39.245 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:01:39.245 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:01:39.245 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:01:39.245 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:01:49.247 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:01:49.247 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:01:49.267 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:01:49.271 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:01:50.202 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:01:50.202 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:01:50.203 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796110 2023-10-20T10:01:50.203 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:01:50.203 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:01:50.204 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:01:50.204 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:02:00.221 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:02:00.222 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:02:00.232 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:02:00.247 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:02:01.138 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:02:01.138 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:02:01.139 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796121 2023-10-20T10:02:01.140 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:02:01.140 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:02:01.140 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:02:01.140 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:02:11.140 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:02:11.141 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:02:11.148 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:02:11.155 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:02:12.132 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:02:12.133 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:02:12.135 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796132 2023-10-20T10:02:12.135 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:02:12.135 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:02:12.136 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:02:12.136 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:02:22.146 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:02:22.147 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:02:22.148 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:02:22.166 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:02:22.584 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:02:22.584+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:02:22.592 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:02:22.596+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:02:23.204 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:02:23.205 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:02:23.205 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796143 2023-10-20T10:02:23.206 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:02:23.206 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:02:23.206 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:02:23.206 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:02:33.208 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:02:33.208 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:02:33.209 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:02:33.215 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:02:34.118 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:02:34.119 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:02:34.121 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796154 2023-10-20T10:02:34.121 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:02:34.121 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:02:34.121 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:02:34.122 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:02:38.553 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:02:38.555+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:02:38.553 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:02:38.555+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T10:02:38.553 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:02:38.555+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:02:38.554 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:02:38.555+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab4f70 2023-10-20T10:02:38.554 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:02:38.555+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T10:02:38.554 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:02:38.555+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:02:38.554 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:02:38.555+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T10:02:38.554 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:02:38.555+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T10:02:38.555 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:02:38.555+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:02:38.555 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:02:38.555+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:02:38.555 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:02:38.555+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2750 2023-10-20T10:02:38.555 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:02:38.555+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2820 2023-10-20T10:02:44.122 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:02:44.124 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:02:44.132 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:02:44.133 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:02:44.909 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:02:44.910 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:02:44.911 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796164 2023-10-20T10:02:44.911 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:02:44.911 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:02:44.911 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:02:44.911 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:02:54.912 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:02:54.916 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:02:54.918 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:02:54.921 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:02:55.778 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:02:55.778 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:02:55.778 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796175 2023-10-20T10:02:55.779 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:02:55.779 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:02:55.779 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:02:55.779 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:03:05.781 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:03:05.781 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:03:05.782 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:03:05.804 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:03:06.660 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:03:06.661 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:03:06.661 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796186 2023-10-20T10:03:06.662 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:03:06.662 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:03:06.662 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:03:06.662 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:03:16.664 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:03:16.665 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:03:16.667 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:03:16.680 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:03:17.599 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:03:17.600 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:03:17.601 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796197 2023-10-20T10:03:17.602 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:03:17.602 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:03:17.602 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:03:17.603 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:03:27.604 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:03:27.604 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:03:27.604 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:03:27.625 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:03:28.720 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:03:28.720 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:03:28.720 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796208 2023-10-20T10:03:28.721 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:03:28.721 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:03:28.721 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:03:28.721 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:03:38.725 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:03:38.731 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:03:38.734 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:03:38.743 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:03:39.819 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:03:39.828 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:03:39.830 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796219 2023-10-20T10:03:39.830 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:03:39.830 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:03:39.830 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:03:39.830 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:03:49.836 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:03:49.837 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:03:49.845 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:03:49.847 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:03:50.742 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:03:50.742 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:03:50.743 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796230 2023-10-20T10:03:50.743 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:03:50.743 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:03:50.743 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:03:50.744 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:04:00.751 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:04:00.752 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:04:00.753 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:04:00.762 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:04:01.642 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:04:01.642 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:04:01.642 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796241 2023-10-20T10:04:01.643 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:04:01.643 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:04:01.643 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:04:01.643 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:04:03.179 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:04:03.178+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:04:03.200 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:04:03.198+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T10:04:03.200 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:04:03.198+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:04:03.206 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:04:03.206+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab4f70 2023-10-20T10:04:03.206 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:04:03.206+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T10:04:03.206 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:04:03.206+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:04:03.207 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:04:03.206+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T10:04:03.207 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:04:03.206+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T10:04:03.207 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:04:03.206+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:04:03.207 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:04:03.206+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:04:03.207 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:04:03.206+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2750 2023-10-20T10:04:11.648 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:04:11.654 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:04:11.654 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:04:11.669 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:04:12.665 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:04:12.665 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:04:12.666 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796252 2023-10-20T10:04:12.666 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:04:12.666 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:04:12.667 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:04:12.667 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:04:22.679 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:04:22.684 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:04:22.687 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:04:22.690 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:04:23.509 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:04:23.510 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:04:23.511 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796263 2023-10-20T10:04:23.511 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:04:23.511 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:04:23.511 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:04:23.511 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:04:33.514 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:04:33.515 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:04:33.516 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:04:33.518 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:04:34.488 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:04:34.488 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:04:34.489 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796274 2023-10-20T10:04:34.489 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:04:34.489 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:04:34.489 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:04:34.490 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:04:44.491 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:04:44.491 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:04:44.491 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:04:44.492 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:04:45.385 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:04:45.386 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:04:45.386 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796285 2023-10-20T10:04:45.387 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:04:45.387 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:04:45.387 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:04:45.387 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:04:48.704 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:04:48.705+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T10:04:48.709 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:04:48.713+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T10:04:48.709 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:04:48.713+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:04:48.710 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:04:48.713+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:04:55.396 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:04:55.397 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:04:55.403 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:04:55.407 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:04:56.310 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:04:56.311 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:04:56.311 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796296 2023-10-20T10:04:56.311 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:04:56.312 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:04:56.312 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:04:56.312 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:05:06.313 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:05:06.314 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:05:06.324 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:05:06.335 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:05:07.316 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:05:07.317 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:05:07.317 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796307 2023-10-20T10:05:07.318 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:05:07.318 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:05:07.318 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:05:07.318 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:05:14.930 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:05:14.928+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:05:14.934 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:05:14.936+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T10:05:14.934 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:05:14.936+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:05:14.940 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:05:14.940+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab4f70 2023-10-20T10:05:14.940 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:05:14.940+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T10:05:14.941 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:05:14.940+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:05:14.941 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:05:14.940+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T10:05:14.941 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:05:14.940+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T10:05:14.941 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:05:14.940+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:05:14.942 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:05:14.940+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:05:17.319 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:05:17.322 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:05:17.324 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:05:17.324 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:05:18.403 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:05:18.403 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:05:18.404 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796318 2023-10-20T10:05:18.404 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:05:18.404 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:05:18.405 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:05:18.405 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:05:28.409 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:05:28.409 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:05:28.413 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:05:28.421 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:05:29.281 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:05:29.281 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:05:29.282 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796329 2023-10-20T10:05:29.282 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:05:29.283 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:05:29.283 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:05:29.283 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:05:39.288 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:05:39.289 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:05:39.291 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:05:39.293 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:05:40.142 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:05:40.143 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:05:40.143 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796340 2023-10-20T10:05:40.143 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:05:40.143 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:05:40.144 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:05:40.144 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:05:50.147 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:05:50.147 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:05:50.147 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:05:50.155 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:05:51.002 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:05:51.002 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:05:51.003 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796351 2023-10-20T10:05:51.003 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:05:51.004 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:05:51.004 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:05:51.004 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:06:01.011 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:06:01.012 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:06:01.014 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:06:01.037 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:06:02.138 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:06:02.144 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:06:02.146 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796362 2023-10-20T10:06:02.146 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:06:02.147 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:06:02.147 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:06:02.147 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:06:12.148 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:06:12.148 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:06:12.154 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:06:12.156 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:06:13.089 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:06:13.089 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:06:13.090 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796373 2023-10-20T10:06:13.090 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:06:13.091 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:06:13.091 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:06:13.091 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:06:23.091 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:06:23.092 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:06:23.092 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:06:23.096 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:06:24.081 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:06:24.082 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:06:24.082 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796384 2023-10-20T10:06:24.082 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:06:24.083 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:06:24.083 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:06:24.083 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:06:34.098 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:06:34.108 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:06:34.113 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:06:34.113 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:06:35.019 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:06:35.019 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:06:35.020 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796395 2023-10-20T10:06:35.020 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:06:35.021 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:06:35.021 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:06:35.021 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:06:45.032 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:06:45.033 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:06:45.036 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:06:45.037 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:06:46.048 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:06:46.048 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:06:46.049 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796406 2023-10-20T10:06:46.049 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:06:46.049 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:06:46.050 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:06:46.050 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:06:56.055 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:06:56.058 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:06:56.060 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:06:56.062 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:06:57.096 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:06:57.098 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:06:57.099 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796417 2023-10-20T10:06:57.099 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:06:57.100 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:06:57.100 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:06:57.100 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:07:07.101 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:07:07.102 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:07:07.103 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:07:07.108 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:07:08.000 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:07:08.006 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:07:08.007 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796428 2023-10-20T10:07:08.007 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:07:08.007 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:07:08.008 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:07:08.008 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:07:18.011 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:07:18.012 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:07:18.012 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:07:18.014 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:07:18.989 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:07:18.993 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:07:18.994 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796438 2023-10-20T10:07:18.995 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:07:18.995 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:07:18.995 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:07:18.996 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:07:29.004 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:07:29.004 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:07:29.005 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:07:29.015 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:07:29.850 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:07:29.858 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:07:29.860 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796449 2023-10-20T10:07:29.860 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:07:29.860 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:07:29.860 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:07:29.861 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:07:38.901 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:07:38.897+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:07:38.902 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:07:38.897+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:07:39.863 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:07:39.863 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:07:39.866 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:07:39.872 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:07:40.276 INFO:tasks.workunit.client.0.smithi150.stderr:2023-10-20T10:07:40.277+0000 7fea4a59c640 0 --2- 172.21.15.150:0/3052311349 >> v2:172.21.15.150:3300/0 conn(0x7fea4401f2e0 0x7fea4401f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-20T10:07:40.967 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:07:40.968 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:07:40.968 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796460 2023-10-20T10:07:40.969 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:07:40.969 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:07:40.969 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:07:40.969 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:07:50.974 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:07:50.983 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:07:50.990 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:07:50.995 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:07:51.951 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:07:51.951 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:07:51.952 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796471 2023-10-20T10:07:51.952 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:07:51.952 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:07:51.952 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:07:51.953 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:07:59.118 INFO:tasks.ceph.osd.2.smithi150.stderr:2023-10-20T10:07:59.121+0000 7f12576b6640 -1 reset not still connected to 0x5635c0e90410 2023-10-20T10:08:01.953 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:08:01.954 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:08:01.954 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:08:01.960 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:08:02.936 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:08:02.937 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:08:02.938 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796482 2023-10-20T10:08:02.938 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:08:02.938 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:08:02.938 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:08:02.939 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:08:12.948 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:08:12.952 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:08:12.956 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:08:12.961 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:08:13.859 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:08:13.859 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:08:13.860 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796493 2023-10-20T10:08:13.860 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:08:13.861 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:08:13.861 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:08:13.861 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:08:14.910 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:08:14.913+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:08:14.922 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:08:14.921+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T10:08:14.922 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:08:14.921+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:08:14.926 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:08:14.929+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab4f70 2023-10-20T10:08:14.926 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:08:14.929+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T10:08:14.926 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:08:14.929+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:08:14.927 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:08:14.929+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T10:08:14.927 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:08:14.929+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T10:08:14.927 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:08:14.929+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:08:14.927 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:08:14.929+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:08:14.927 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:08:14.929+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2750 2023-10-20T10:08:14.928 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:08:14.929+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2820 2023-10-20T10:08:23.862 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:08:23.864 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:08:23.867 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:08:23.878 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:08:24.859 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:08:24.859 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:08:24.861 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796504 2023-10-20T10:08:24.861 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:08:24.861 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:08:24.861 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:08:24.862 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:08:34.866 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:08:34.867 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:08:34.867 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:08:34.870 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:08:35.738 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:08:35.739 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:08:35.739 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796515 2023-10-20T10:08:35.739 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:08:35.739 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:08:35.740 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:08:35.740 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:08:45.741 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:08:45.742 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:08:45.753 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:08:45.755 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:08:46.637 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:08:46.637 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:08:46.638 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796526 2023-10-20T10:08:46.638 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:08:46.638 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:08:46.638 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:08:46.639 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:08:56.640 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:08:56.646 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:08:56.646 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:08:56.648 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:08:57.555 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:08:57.555 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:08:57.556 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796537 2023-10-20T10:08:57.556 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:08:57.556 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:08:57.556 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:08:57.556 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:09:07.558 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:09:07.560 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:09:07.566 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:09:07.566 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:09:08.472 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:09:08.473 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:09:08.474 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796548 2023-10-20T10:09:08.474 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:09:08.475 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:09:08.475 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:09:08.475 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:09:16.989 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:09:16.987+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T10:09:16.989 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:09:16.987+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:09:16.990 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:09:16.987+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:09:18.489 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:09:18.493 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:09:18.505 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:09:18.507 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:09:18.920 INFO:tasks.workunit.client.0.smithi150.stderr:2023-10-20T10:09:18.923+0000 7f16dd226640 0 --2- 172.21.15.150:0/4122797605 >> v2:172.21.15.150:3300/0 conn(0x7f16d8090a00 0x7f16d8128980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-20T10:09:19.779 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:09:19.779 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:09:19.780 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796559 2023-10-20T10:09:19.780 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:09:19.780 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:09:19.780 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:09:19.780 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:09:29.786 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:09:29.786 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:09:29.787 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:09:29.800 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:09:30.844 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:09:30.844 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:09:30.845 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796570 2023-10-20T10:09:30.845 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:09:30.845 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:09:30.846 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:09:30.846 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:09:40.846 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:09:40.850 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:09:40.855 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:09:40.856 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:09:41.859 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:09:41.859 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:09:41.860 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796581 2023-10-20T10:09:41.860 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:09:41.861 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:09:41.861 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:09:41.861 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:09:41.983 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:09:41.979+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:09:41.993 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:09:41.987+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T10:09:41.993 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:09:41.987+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:09:41.994 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:09:41.991+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T10:09:41.994 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:09:41.991+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:09:41.994 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:09:41.991+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T10:09:41.995 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:09:41.991+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:09:41.995 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:09:41.991+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:09:51.862 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:09:51.862 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:09:51.865 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:09:51.879 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:09:52.839 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:09:52.839 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:09:52.840 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796592 2023-10-20T10:09:52.840 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:09:52.840 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:09:52.841 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:09:52.841 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:09:59.539 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:09:59.539+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:09:59.549 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:09:59.551+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T10:09:59.550 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:09:59.551+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:09:59.560 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:09:59.559+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab4f70 2023-10-20T10:09:59.560 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:09:59.559+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T10:09:59.561 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:09:59.559+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:09:59.561 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:09:59.559+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T10:09:59.565 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:09:59.567+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T10:09:59.565 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:09:59.567+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:09:59.565 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:09:59.567+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:09:59.565 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:09:59.567+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2750 2023-10-20T10:10:02.842 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:10:02.843 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:10:02.844 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:10:02.866 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:10:03.794 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:10:03.794 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:10:03.795 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796603 2023-10-20T10:10:03.795 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:10:03.795 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:10:03.796 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:10:03.796 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:10:13.797 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:10:13.798 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:10:13.798 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:10:13.813 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:10:14.691 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:10:14.691 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:10:14.692 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796614 2023-10-20T10:10:14.692 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:10:14.693 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:10:14.693 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:10:14.693 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:10:16.150 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:10:16.150+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:10:16.151 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:10:16.150+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:10:16.158 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:10:16.158+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:10:24.695 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:10:24.696 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:10:24.700 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:10:24.700 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:10:25.629 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:10:25.629 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:10:25.630 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796625 2023-10-20T10:10:25.630 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:10:25.630 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:10:25.630 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:10:25.630 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:10:34.641 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:10:34.634+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:10:34.642 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:10:34.638+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:10:34.656 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:10:34.654+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:10:35.599 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:10:35.598+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:10:35.627 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:10:35.626+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:10:35.631 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:10:35.643 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:10:35.652 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:10:35.654 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:10:36.565 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:10:36.565 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:10:36.566 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796636 2023-10-20T10:10:36.566 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:10:36.566 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:10:36.566 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:10:36.567 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:10:46.161 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:10:46.162+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:10:46.162 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:10:46.162+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T10:10:46.162 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:10:46.162+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:10:46.166 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:10:46.166+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab4f70 2023-10-20T10:10:46.167 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:10:46.166+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T10:10:46.167 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:10:46.166+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:10:46.167 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:10:46.166+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T10:10:46.167 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:10:46.166+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T10:10:46.168 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:10:46.166+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:10:46.168 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:10:46.166+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:10:46.168 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:10:46.166+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2750 2023-10-20T10:10:46.168 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:10:46.166+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2820 2023-10-20T10:10:46.578 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:10:46.579 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:10:46.584 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:10:46.586 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:10:47.537 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:10:47.537 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:10:47.537 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796647 2023-10-20T10:10:47.538 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:10:47.538 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:10:47.538 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:10:47.538 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:10:57.546 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:10:57.547 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:10:57.547 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:10:57.547 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:10:58.491 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:10:58.491 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:10:58.492 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796658 2023-10-20T10:10:58.492 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:10:58.492 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:10:58.492 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:10:58.493 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:11:08.494 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:11:08.500 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:11:08.506 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:11:08.517 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:11:09.396 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:11:09.398 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:11:09.399 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796669 2023-10-20T10:11:09.400 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:11:09.400 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:11:09.400 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:11:09.400 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:11:19.402 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:11:19.403 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:11:19.403 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:11:19.407 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:11:20.336 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:11:20.337 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:11:20.346 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796680 2023-10-20T10:11:20.347 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:11:20.347 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:11:20.347 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:11:20.347 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:11:30.348 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:11:30.354 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:11:30.356 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:11:30.366 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:11:31.269 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:11:31.269 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:11:31.270 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796691 2023-10-20T10:11:31.270 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:11:31.271 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:11:31.271 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:11:31.271 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:11:36.729 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:11:36.733+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:11:36.730 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:11:36.733+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:11:36.735 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:11:36.737+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T10:11:36.735 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:11:36.737+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:11:36.741 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:11:36.745+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T10:11:36.746 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:11:36.749+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T10:11:36.747 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:11:36.749+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:11:36.747 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:11:36.749+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:11:41.274 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:11:41.277 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:11:41.281 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:11:41.289 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:11:42.205 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:11:42.206 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:11:42.206 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796702 2023-10-20T10:11:42.207 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:11:42.207 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:11:42.207 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:11:42.207 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:11:44.960 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:11:44.961+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:11:44.969 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:11:44.973+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T10:11:44.969 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:11:44.973+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:11:44.986 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:11:44.989+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab4f70 2023-10-20T10:11:44.986 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:11:44.989+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T10:11:44.986 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:11:44.989+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:11:44.987 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:11:44.989+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T10:11:44.987 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:11:44.989+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T10:11:44.987 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:11:44.989+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:11:44.987 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:11:44.989+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:11:52.209 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:11:52.209 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:11:52.210 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:11:52.214 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:11:53.053 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:11:53.053 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:11:53.053 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796713 2023-10-20T10:11:53.054 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:11:53.054 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:11:53.054 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:11:53.054 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:12:03.056 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:12:03.057 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:12:03.057 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:12:03.069 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:12:03.911 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:12:03.912 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:12:03.912 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796723 2023-10-20T10:12:03.912 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:12:03.913 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:12:03.913 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:12:03.913 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:12:13.914 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:12:13.915 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:12:13.916 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:12:13.935 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:12:14.796 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:12:14.797 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:12:14.797 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796734 2023-10-20T10:12:14.798 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:12:14.798 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:12:14.798 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:12:14.798 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:12:24.803 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:12:24.804 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:12:24.811 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:12:24.813 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:12:25.686 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:12:25.686 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:12:25.688 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796745 2023-10-20T10:12:25.688 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:12:25.688 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:12:25.688 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:12:25.688 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:12:35.696 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:12:35.697 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:12:35.698 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:12:35.708 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:12:36.593 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:12:36.597 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:12:36.598 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796756 2023-10-20T10:12:36.599 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:12:36.599 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:12:36.599 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:12:36.599 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:12:46.601 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:12:46.602 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:12:46.602 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:12:46.611 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:12:47.647 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:12:47.648 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:12:47.648 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796767 2023-10-20T10:12:47.648 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:12:47.649 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:12:47.649 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:12:47.649 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:12:48.219 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:12:48.220+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T10:12:48.220 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:12:48.220+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:12:48.220 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:12:48.220+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T10:12:48.220 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:12:48.220+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:12:48.232 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:12:48.228+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:12:53.272 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:12:53.275+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:12:57.651 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:12:57.652 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:12:57.652 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:12:57.652 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:12:58.543 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:12:58.544 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:12:58.545 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796778 2023-10-20T10:12:58.545 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:12:58.545 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:12:58.545 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:12:58.546 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:13:08.547 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:13:08.547 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:13:08.548 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:13:08.548 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:13:09.603 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:13:09.603 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:13:09.604 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796789 2023-10-20T10:13:09.604 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:13:09.604 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:13:09.605 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:13:09.605 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:13:19.606 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:13:19.610 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:13:19.620 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:13:19.622 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:13:20.576 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:13:20.577 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:13:20.578 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796800 2023-10-20T10:13:20.578 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:13:20.578 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:13:20.579 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:13:20.579 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:13:30.585 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:13:30.585 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:13:30.585 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:13:30.588 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:13:31.407 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:13:31.408 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:13:31.408 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796811 2023-10-20T10:13:31.408 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:13:31.408 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:13:31.409 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:13:31.409 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:13:41.414 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:13:41.414 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:13:41.424 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:13:41.426 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:13:42.403 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:13:42.403 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:13:42.403 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796822 2023-10-20T10:13:42.404 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:13:42.404 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:13:42.404 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:13:42.404 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:13:45.213 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:13:45.214+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:13:52.410 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:13:52.413 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:13:52.416 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:13:52.421 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:13:53.521 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:13:53.521 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:13:53.522 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796833 2023-10-20T10:13:53.522 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:13:53.522 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:13:53.522 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:13:53.523 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:14:03.534 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:14:03.534 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:14:03.534 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:14:03.556 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:14:04.501 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:14:04.501 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:14:04.502 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796844 2023-10-20T10:14:04.502 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:14:04.502 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:14:04.503 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:14:04.503 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:14:07.749 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:14:07.750+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:14:07.750 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:14:07.750+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:14:08.948 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:14:08.942+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:14:08.948 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:14:08.946+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T10:14:08.949 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:14:08.946+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:14:08.952 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:14:08.950+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab4f70 2023-10-20T10:14:08.952 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:14:08.950+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T10:14:08.953 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:14:08.950+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:14:08.953 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:14:08.950+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T10:14:08.953 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:14:08.950+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T10:14:08.953 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:14:08.950+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:14:08.953 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:14:08.950+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:14:08.961 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:14:08.962+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2750 2023-10-20T10:14:12.834 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:14:12.826+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:14:12.848 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:14:12.838+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:14:14.504 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:14:14.509 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:14:14.513 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:14:14.513 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:14:15.357 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:14:15.357 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:14:15.358 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796855 2023-10-20T10:14:15.358 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:14:15.358 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:14:15.359 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:14:15.359 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:14:20.637 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:14:20.621+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:14:20.640 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:14:20.637+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:14:20.653 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:14:20.645+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:14:25.360 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:14:25.360 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:14:25.363 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:14:25.371 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:14:26.176 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:14:26.176 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:14:26.177 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796866 2023-10-20T10:14:26.177 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:14:26.177 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:14:26.178 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:14:26.178 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:14:30.214 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:14:30.213+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:14:30.244 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:14:30.241+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T10:14:30.244 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:14:30.241+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:14:30.245 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:14:30.241+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:14:36.182 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:14:36.183 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:14:36.183 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:14:36.191 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:14:37.288 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:14:37.291 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:14:37.292 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796877 2023-10-20T10:14:37.292 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:14:37.292 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:14:37.292 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:14:37.293 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:14:47.294 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:14:47.294 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:14:47.295 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:14:47.321 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:14:48.362 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:14:48.363 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:14:48.363 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796888 2023-10-20T10:14:48.363 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:14:48.363 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:14:48.364 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:14:48.364 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:14:58.365 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:14:58.365 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:14:58.371 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:14:58.376 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:14:59.156 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:14:59.156 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:14:59.157 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796899 2023-10-20T10:14:59.157 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:14:59.158 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:14:59.158 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:14:59.158 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:15:06.834 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:15:06.836+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:15:06.845 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:15:06.844+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T10:15:06.845 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:15:06.844+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:15:06.849 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:15:06.852+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T10:15:06.861 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:15:06.864+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T10:15:06.861 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:15:06.864+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:15:06.862 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:15:06.864+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:15:09.159 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:15:09.160 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:15:09.161 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:15:09.174 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:15:10.111 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:15:10.111 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:15:10.112 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796910 2023-10-20T10:15:10.112 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:15:10.113 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:15:10.113 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:15:10.113 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:15:20.122 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:15:20.123 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:15:20.129 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:15:20.135 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:15:21.147 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:15:21.148 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:15:21.149 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796921 2023-10-20T10:15:21.149 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:15:21.150 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ -n 172.21.15.150:0/3774459303 ]] 2023-10-20T10:15:21.150 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 172.21.15.150:0/3774459303 != \1\7\2\.\2\1\.\1\5\.\1\5\0\:\0\/\3\7\7\4\4\5\9\3\0\3 ]] 2023-10-20T10:15:21.150 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:15:25.147 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:15:25.148+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:15:25.150 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:15:25.152+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:15:31.154 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-20T10:15:31.154 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq '.active_clients[]' 2023-10-20T10:15:31.158 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph mgr dump 2023-10-20T10:15:31.159 INFO:tasks.workunit.client.0.smithi150.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-20T10:15:32.230 INFO:tasks.workunit.client.0.smithi150.stderr:+ CLIENT_ADDR=172.21.15.150:0/3774459303 2023-10-20T10:15:32.236 INFO:tasks.workunit.client.0.smithi150.stderr:++ date +%s 2023-10-20T10:15:32.237 INFO:tasks.workunit.client.0.smithi150.stderr:+ CURRENT_TIME=1697796932 2023-10-20T10:15:32.237 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-20T10:15:32.237 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i = 1 )) 2023-10-20T10:15:32.238 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 24 )) 2023-10-20T10:15:32.238 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-20T10:15:32.469 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:15:32.472+0000 7f289f426640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-20T10:15:32.470 INFO:tasks.workunit.client.0.smithi150.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-20T10:15:32.476 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:15:42.478 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T10:15:42.478 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 24 )) 2023-10-20T10:15:42.478 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-20T10:15:42.531 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:15:42.531+0000 7f289f426640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-20T10:15:42.532 INFO:tasks.workunit.client.0.smithi150.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-20T10:15:42.540 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:15:52.541 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T10:15:52.542 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 24 )) 2023-10-20T10:15:52.542 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-20T10:15:52.619 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:15:52.619+0000 7f289f426640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-20T10:15:52.619 INFO:tasks.workunit.client.0.smithi150.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-20T10:15:52.640 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:16:02.145 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:16:02.147+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:16:02.146 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:16:02.147+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:16:02.642 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T10:16:02.643 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 24 )) 2023-10-20T10:16:02.643 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-20T10:16:02.730 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:16:02.731+0000 7f289f426640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-20T10:16:02.730 INFO:tasks.workunit.client.0.smithi150.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-20T10:16:02.739 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:16:12.743 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T10:16:12.743 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 24 )) 2023-10-20T10:16:12.744 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-20T10:16:12.818 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:16:12.819+0000 7f289f426640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-20T10:16:12.818 INFO:tasks.workunit.client.0.smithi150.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-20T10:16:12.827 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:16:22.829 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T10:16:22.829 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 24 )) 2023-10-20T10:16:22.829 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-20T10:16:22.925 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:16:22.923+0000 7f289f426640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-20T10:16:22.925 INFO:tasks.workunit.client.0.smithi150.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-20T10:16:22.932 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:16:32.933 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T10:16:32.934 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 24 )) 2023-10-20T10:16:32.934 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-20T10:16:33.007 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:16:33.006+0000 7f289f426640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-20T10:16:33.008 INFO:tasks.workunit.client.0.smithi150.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-20T10:16:33.017 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:16:43.019 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T10:16:43.019 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 24 )) 2023-10-20T10:16:43.019 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-20T10:16:43.139 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:16:43.138+0000 7f289f426640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-20T10:16:43.139 INFO:tasks.workunit.client.0.smithi150.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-20T10:16:43.142 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:16:50.631 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:16:50.630+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:16:50.647 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:16:50.646+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:16:50.655 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:16:50.654+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:16:53.143 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T10:16:53.144 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 24 )) 2023-10-20T10:16:53.144 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-20T10:16:53.231 INFO:tasks.workunit.client.0.smithi150.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-20T10:16:53.235 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:16:53.230+0000 7f289f426640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-20T10:16:53.237 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:17:03.239 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T10:17:03.239 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 24 )) 2023-10-20T10:17:03.239 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-20T10:17:03.321 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:17:03.318+0000 7f289f426640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-20T10:17:03.321 INFO:tasks.workunit.client.0.smithi150.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-20T10:17:03.325 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:17:13.327 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T10:17:13.327 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 24 )) 2023-10-20T10:17:13.327 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-20T10:17:13.432 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:17:13.429+0000 7f289f426640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-20T10:17:13.433 INFO:tasks.workunit.client.0.smithi150.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-20T10:17:13.438 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:17:23.440 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T10:17:23.440 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 24 )) 2023-10-20T10:17:23.440 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-20T10:17:23.551 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:17:23.549+0000 7f289f426640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-20T10:17:23.552 INFO:tasks.workunit.client.0.smithi150.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-20T10:17:23.557 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:17:32.546 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:17:32.541+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:17:32.547 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:17:32.545+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:17:32.564 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:17:32.557+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:17:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T10:17:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 24 )) 2023-10-20T10:17:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-20T10:17:33.661 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:17:33.657+0000 7f289f426640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-20T10:17:33.661 INFO:tasks.workunit.client.0.smithi150.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-20T10:17:33.669 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:17:43.686 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T10:17:43.687 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 24 )) 2023-10-20T10:17:43.687 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-20T10:17:43.770 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:17:43.773+0000 7f289f426640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-20T10:17:43.771 INFO:tasks.workunit.client.0.smithi150.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-20T10:17:43.777 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:17:53.778 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T10:17:53.779 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 24 )) 2023-10-20T10:17:53.779 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-20T10:17:53.849 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:17:53.853+0000 7f289f426640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-20T10:17:53.849 INFO:tasks.workunit.client.0.smithi150.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-20T10:17:53.857 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:18:03.862 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T10:18:03.862 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 24 )) 2023-10-20T10:18:03.862 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-20T10:18:03.986 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:18:03.988+0000 7f289f426640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-20T10:18:03.987 INFO:tasks.workunit.client.0.smithi150.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-20T10:18:03.996 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:18:13.997 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T10:18:13.997 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 24 )) 2023-10-20T10:18:13.998 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-20T10:18:14.074 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:18:14.076+0000 7f289f426640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-20T10:18:14.074 INFO:tasks.workunit.client.0.smithi150.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-20T10:18:14.085 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:18:24.087 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T10:18:24.087 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 24 )) 2023-10-20T10:18:24.087 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-20T10:18:24.204 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:18:24.204+0000 7f289f426640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-20T10:18:24.205 INFO:tasks.workunit.client.0.smithi150.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-20T10:18:24.212 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:18:34.213 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T10:18:34.213 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 24 )) 2023-10-20T10:18:34.213 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-20T10:18:34.346 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:18:34.348+0000 7f289f426640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-20T10:18:34.347 INFO:tasks.workunit.client.0.smithi150.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-20T10:18:34.362 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:18:44.363 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T10:18:44.364 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 24 )) 2023-10-20T10:18:44.364 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-20T10:18:44.469 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:18:44.468+0000 7f289f426640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-20T10:18:44.469 INFO:tasks.workunit.client.0.smithi150.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-20T10:18:44.489 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:18:54.499 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T10:18:54.500 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 24 )) 2023-10-20T10:18:54.500 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-20T10:18:54.569 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:18:54.567+0000 7f289f426640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-20T10:18:54.569 INFO:tasks.workunit.client.0.smithi150.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-20T10:18:54.586 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:19:02.028 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:19:02.027+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:19:04.588 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T10:19:04.588 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 24 )) 2023-10-20T10:19:04.588 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-20T10:19:04.711 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:19:04.711+0000 7f289f426640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-20T10:19:04.711 INFO:tasks.workunit.client.0.smithi150.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-20T10:19:04.716 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:19:14.717 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T10:19:14.718 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 24 )) 2023-10-20T10:19:14.718 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-20T10:19:14.785 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:19:14.787+0000 7f289f426640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-20T10:19:14.786 INFO:tasks.workunit.client.0.smithi150.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-20T10:19:14.797 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:19:24.798 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T10:19:24.799 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 24 )) 2023-10-20T10:19:24.799 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-20T10:19:24.888 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:19:24.887+0000 7f289f426640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-20T10:19:24.889 INFO:tasks.workunit.client.0.smithi150.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-20T10:19:24.908 INFO:tasks.workunit.client.0.smithi150.stderr:+ sleep 10 2023-10-20T10:19:32.779 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:19:32.775+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:19:32.781 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:19:32.779+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:19:34.909 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i++ )) 2023-10-20T10:19:34.910 INFO:tasks.workunit.client.0.smithi150.stderr:+ (( i <= 24 )) 2023-10-20T10:19:34.910 INFO:tasks.workunit.client.0.smithi150.stderr:+ rbd mirror snapshot schedule ls -p rbd --image image1 2023-10-20T10:19:34.922 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep 'every 2m' 2023-10-20T10:19:35.263 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:19:35.263+0000 7f289f426640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-20T10:19:35.264 INFO:tasks.workunit.client.0.smithi150.stderr:rbd: rbd mirror snapshot schedule list failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-20T10:19:35.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T10:19:35.278 INFO:tasks.workunit:Stopping ['rbd/rbd_support_module_recovery.sh'] on client.0... 2023-10-20T10:19:35.279 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-20T10:19:35.980 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi150 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-20T10:19:36.052 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=6223904771914abea1419e2dce9f3d24 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7f103c7cad2e64eeb2f3003665c1f2f43a9381ab/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi150 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-20T10:19:36.054 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-20T10:19:36.118 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-20T10:19:36.127 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-20T10:19:36.128 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:19:36.666 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:19:36.667 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:19:36.703 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2102,"stamp":"2023-10-20T10:19:35.201479+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27082912259,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1787635,"num_read_kb":6909581,"num_write":1746883,"num_write_kb":6985783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68735,"ondisk_log_size":68735,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18145988,"kb_used_data":13970796,"kb_used_omap":2805,"kb_used_meta":4172362,"kb_avail":264969532,"statfs":{"total":289910292480,"available":271328800768,"internally_reserved":0,"allocated":14306095104,"data_stored":27686083302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2873140,"internal_metadata":4272498892},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,10,7,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22061056,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5204,"num_read_kb":20752,"num_write":5345,"num_write_kb":21380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002058"},"pg_stats":[{"pgid":"2.7","version":"122'8915","reported_seq":13400,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:32.985139+0000","last_change":"2023-10-20T09:13:38.742871+0000","last_active":"2023-10-20T10:19:32.985139+0000","last_peered":"2023-10-20T10:19:32.985139+0000","last_clean":"2023-10-20T10:19:32.985139+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:19:32.985139+0000","last_undegraded":"2023-10-20T10:19:32.985139+0000","last_fullsized":"2023-10-20T10:19:32.985139+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":8915,"log_dups_size":0,"ondisk_log_size":8915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:18:55.451570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"122'17523","reported_seq":27498,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:33.009850+0000","last_change":"2023-10-20T09:13:38.742974+0000","last_active":"2023-10-20T10:19:33.009850+0000","last_peered":"2023-10-20T10:19:33.009850+0000","last_clean":"2023-10-20T10:19:33.009850+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:19:33.009850+0000","last_undegraded":"2023-10-20T10:19:33.009850+0000","last_fullsized":"2023-10-20T10:19:33.009850+0000","mapping_epoch":13,"log_start":"92'7500","ondisk_log_start":"92'7500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:38:55.327279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"101'10","reported_seq":180,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-20T10:18:38.557537+0000","last_change":"2023-10-20T09:13:38.742635+0000","last_active":"2023-10-20T10:18:38.557537+0000","last_peered":"2023-10-20T10:18:38.557537+0000","last_clean":"2023-10-20T10:18:38.557537+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:18:38.557537+0000","last_undegraded":"2023-10-20T10:18:38.557537+0000","last_fullsized":"2023-10-20T10:18:38.557537+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:42:15.061080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"122'8880","reported_seq":14139,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:32.985180+0000","last_change":"2023-10-20T09:13:38.742777+0000","last_active":"2023-10-20T10:19:32.985180+0000","last_peered":"2023-10-20T10:19:32.985180+0000","last_clean":"2023-10-20T10:19:32.985180+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:19:32.985180+0000","last_undegraded":"2023-10-20T10:19:32.985180+0000","last_fullsized":"2023-10-20T10:19:32.985180+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":8880,"log_dups_size":0,"ondisk_log_size":8880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:36:13.287316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"122'451937","reported_seq":897491,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:33.130321+0000","last_change":"2023-10-20T09:46:02.941197+0000","last_active":"2023-10-20T10:19:33.130321+0000","last_peered":"2023-10-20T10:19:33.130321+0000","last_clean":"2023-10-20T10:19:33.130321+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:19:33.130321+0000","last_undegraded":"2023-10-20T10:19:33.130321+0000","last_fullsized":"2023-10-20T10:19:33.130321+0000","mapping_epoch":16,"log_start":"121'441930","ondisk_log_start":"121'441930","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T19:45:00.379140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":6959030272,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":448867,"num_read_kb":1766264,"num_write":448417,"num_write_kb":1793668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"116'34","reported_seq":269,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-20T10:18:37.869240+0000","last_change":"2023-10-20T09:13:38.741487+0000","last_active":"2023-10-20T10:18:37.869240+0000","last_peered":"2023-10-20T10:18:37.869240+0000","last_clean":"2023-10-20T10:18:37.869240+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:18:37.869240+0000","last_undegraded":"2023-10-20T10:18:37.869240+0000","last_fullsized":"2023-10-20T10:18:37.869240+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T15:48:03.587791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"122'419113","reported_seq":832320,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:33.716764+0000","last_change":"2023-10-20T09:46:02.833709+0000","last_active":"2023-10-20T10:19:33.716764+0000","last_peered":"2023-10-20T10:19:33.716764+0000","last_clean":"2023-10-20T10:19:33.716764+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:19:33.716764+0000","last_undegraded":"2023-10-20T10:19:33.716764+0000","last_fullsized":"2023-10-20T10:19:33.716764+0000","mapping_epoch":16,"log_start":"121'409039","ondisk_log_start":"121'409039","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T20:34:58.352460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6451920896,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":416290,"num_read_kb":1635984,"num_write":415820,"num_write_kb":1663280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"120'309","reported_seq":6148,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:30.320070+0000","last_change":"2023-10-20T09:13:38.740624+0000","last_active":"2023-10-20T10:19:30.320070+0000","last_peered":"2023-10-20T10:19:30.320070+0000","last_clean":"2023-10-20T10:19:30.320070+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:19:30.320070+0000","last_undegraded":"2023-10-20T10:19:30.320070+0000","last_fullsized":"2023-10-20T10:19:30.320070+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:21:43.433612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"122'445099","reported_seq":884412,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:33.121072+0000","last_change":"2023-10-20T09:46:02.852528+0000","last_active":"2023-10-20T10:19:33.121072+0000","last_peered":"2023-10-20T10:19:33.121072+0000","last_clean":"2023-10-20T10:19:33.121072+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:19:33.121072+0000","last_undegraded":"2023-10-20T10:19:33.121072+0000","last_fullsized":"2023-10-20T10:19:33.121072+0000","mapping_epoch":16,"log_start":"121'435034","ondisk_log_start":"121'435034","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:22:55.065960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":6852706304,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":442651,"num_read_kb":1740136,"num_write":441562,"num_write_kb":1766248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"120'288","reported_seq":5999,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:30.330960+0000","last_change":"2023-10-20T09:13:38.738881+0000","last_active":"2023-10-20T10:19:30.330960+0000","last_peered":"2023-10-20T10:19:30.330960+0000","last_clean":"2023-10-20T10:19:30.330960+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:19:30.330960+0000","last_undegraded":"2023-10-20T10:19:30.330960+0000","last_fullsized":"2023-10-20T10:19:30.330960+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:59:24.678918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"122'443544","reported_seq":881435,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:33.711766+0000","last_change":"2023-10-20T09:46:03.048518+0000","last_active":"2023-10-20T10:19:33.711766+0000","last_peered":"2023-10-20T10:19:33.711766+0000","last_clean":"2023-10-20T10:19:33.711766+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:19:33.711766+0000","last_undegraded":"2023-10-20T10:19:33.711766+0000","last_fullsized":"2023-10-20T10:19:33.711766+0000","mapping_epoch":16,"log_start":"121'433524","ondisk_log_start":"121'433524","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:07:14.831718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6818660371,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":441156,"num_read_kb":1734476,"num_write":440070,"num_write_kb":1760274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"116'13","reported_seq":176,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-20T10:18:38.557492+0000","last_change":"2023-10-20T09:13:38.738881+0000","last_active":"2023-10-20T10:18:38.557492+0000","last_peered":"2023-10-20T10:18:38.557492+0000","last_clean":"2023-10-20T10:18:38.557492+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:18:38.557492+0000","last_undegraded":"2023-10-20T10:18:38.557492+0000","last_fullsized":"2023-10-20T10:18:38.557492+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:33:03.535449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":280,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-20T10:17:58.370519+0000","last_change":"2023-10-20T09:13:35.016203+0000","last_active":"2023-10-20T10:17:58.370519+0000","last_peered":"2023-10-20T10:17:58.370519+0000","last_clean":"2023-10-20T10:17:58.370519+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:17:58.370519+0000","last_undegraded":"2023-10-20T10:17:58.370519+0000","last_fullsized":"2023-10-20T10:17:58.370519+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:33.563853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:33.563853+0000","last_clean_scrub_stamp":"2023-10-20T09:13:33.563853+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:44:23.181197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27082317843,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1748964,"num_read_kb":6876860,"num_write":1745869,"num_write_kb":6983470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14301831168,"data_stored":27684585510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40166,"ondisk_log_size":40166,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":28472,"ondisk_log_size":28472,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739163,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8485072,"kb_used_data":6983880,"kb_used_omap":1430,"kb_used_meta":1499753,"kb_avail":85886768,"statfs":{"total":96636764160,"available":87948050432,"internally_reserved":0,"allocated":7151493120,"data_stored":13842054703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1465289,"internal_metadata":1535747127},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:18:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":0.60699999999999998,"15min":0.45000000000000001},"min":{"1min":0.16700000000000001,"5min":0.16300000000000001,"15min":0.14699999999999999},"max":{"1min":6.8419999999999996,"5min":8.2449999999999992,"15min":8.2449999999999992},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.55000000000000004,"15min":0.48299999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.8239999999999998,"5min":6.8239999999999998,"15min":6.8239999999999998},"last":0.215}]},{"osd":1,"last update":"Fri Oct 20 10:18:57 2023","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":0.72699999999999998,"15min":0.71899999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.187},"max":{"1min":8.282,"5min":8.282,"15min":35.670000000000002},"last":0.248},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.47699999999999998,"15min":0.63200000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":3.6619999999999999,"5min":4.0629999999999997,"15min":34.948999999999998},"last":0.47599999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739163,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4567848,"kb_used_data":3425248,"kb_used_omap":676,"kb_used_meta":1141915,"kb_avail":89803992,"statfs":{"total":96636764160,"available":91959287808,"internally_reserved":0,"allocated":3507453952,"data_stored":6771677603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1169321547},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.47699999999999998,"15min":0.443},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.157},"max":{"1min":4.0209999999999999,"5min":4.0209999999999999,"15min":4.7039999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.27700000000000002,"5min":0.45200000000000001,"15min":0.437},"min":{"1min":0.20000000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.376,"5min":5.8479999999999999,"15min":7.7510000000000003},"last":0.22}]},{"osd":2,"last update":"Fri Oct 20 10:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.40300000000000002,"15min":0.41799999999999998},"min":{"1min":0.20799999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.60699999999999998,"5min":3.657,"15min":4.8920000000000003},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.45100000000000001,"15min":0.42599999999999999},"min":{"1min":0.26000000000000001,"5min":0.23799999999999999,"15min":0.215},"max":{"1min":0.64000000000000001,"5min":3.6869999999999998,"15min":4.9279999999999999},"last":0.372}]}]},{"osd":0,"up_from":8,"seq":34359739163,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5093068,"kb_used_data":3561668,"kb_used_omap":698,"kb_used_meta":1530693,"kb_avail":89278772,"statfs":{"total":96636764160,"available":91421462528,"internally_reserved":0,"allocated":3647148032,"data_stored":7072350996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":1567430218},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,5,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:18:38 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":1.1799999999999999,"15min":0.72399999999999998},"min":{"1min":0.188,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":1.8240000000000001,"5min":81.197000000000003,"15min":81.197000000000003},"last":0.224},{"interface":"front","average":{"1min":0.27400000000000002,"5min":1.206,"15min":0.67200000000000004},"min":{"1min":0.20399999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.35499999999999998,"5min":80.334999999999994,"15min":80.334999999999994},"last":0.26100000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:18:38 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.61199999999999999,"15min":0.50900000000000001},"min":{"1min":0.23899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":11.176,"5min":11.176,"15min":11.176},"last":0.31},{"interface":"front","average":{"1min":1.1950000000000001,"5min":0.70999999999999996,"15min":0.624},"min":{"1min":0.26900000000000002,"5min":0.20000000000000001,"15min":0.187},"max":{"1min":11.093,"5min":11.093,"15min":11.093},"last":0.29199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3645353984,"data_stored":7071457280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3505733632,"data_stored":6771007507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7150743552,"data_stored":13842120723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:19:36.704 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:19:37.287 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:19:37.288 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:19:37.341 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2103,"stamp":"2023-10-20T10:19:37.201832+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27082912259,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1787635,"num_read_kb":6909581,"num_write":1746883,"num_write_kb":6985783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68735,"ondisk_log_size":68735,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18145988,"kb_used_data":13970796,"kb_used_omap":2805,"kb_used_meta":4172362,"kb_avail":264969532,"statfs":{"total":289910292480,"available":271328800768,"internally_reserved":0,"allocated":14306095104,"data_stored":27686083302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2873140,"internal_metadata":4272498892},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,10,7,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17686528,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4236,"num_read_kb":16892,"num_write":4292,"num_write_kb":17168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002083"},"pg_stats":[{"pgid":"2.7","version":"122'8915","reported_seq":13400,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:32.985139+0000","last_change":"2023-10-20T09:13:38.742871+0000","last_active":"2023-10-20T10:19:32.985139+0000","last_peered":"2023-10-20T10:19:32.985139+0000","last_clean":"2023-10-20T10:19:32.985139+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:19:32.985139+0000","last_undegraded":"2023-10-20T10:19:32.985139+0000","last_fullsized":"2023-10-20T10:19:32.985139+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":8915,"log_dups_size":0,"ondisk_log_size":8915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:18:55.451570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"122'17523","reported_seq":27498,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:33.009850+0000","last_change":"2023-10-20T09:13:38.742974+0000","last_active":"2023-10-20T10:19:33.009850+0000","last_peered":"2023-10-20T10:19:33.009850+0000","last_clean":"2023-10-20T10:19:33.009850+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:19:33.009850+0000","last_undegraded":"2023-10-20T10:19:33.009850+0000","last_fullsized":"2023-10-20T10:19:33.009850+0000","mapping_epoch":13,"log_start":"92'7500","ondisk_log_start":"92'7500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:38:55.327279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"101'10","reported_seq":180,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-20T10:18:38.557537+0000","last_change":"2023-10-20T09:13:38.742635+0000","last_active":"2023-10-20T10:18:38.557537+0000","last_peered":"2023-10-20T10:18:38.557537+0000","last_clean":"2023-10-20T10:18:38.557537+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:18:38.557537+0000","last_undegraded":"2023-10-20T10:18:38.557537+0000","last_fullsized":"2023-10-20T10:18:38.557537+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:42:15.061080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"122'8880","reported_seq":14139,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:32.985180+0000","last_change":"2023-10-20T09:13:38.742777+0000","last_active":"2023-10-20T10:19:32.985180+0000","last_peered":"2023-10-20T10:19:32.985180+0000","last_clean":"2023-10-20T10:19:32.985180+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:19:32.985180+0000","last_undegraded":"2023-10-20T10:19:32.985180+0000","last_fullsized":"2023-10-20T10:19:32.985180+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":8880,"log_dups_size":0,"ondisk_log_size":8880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:36:13.287316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"122'451937","reported_seq":897491,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:33.130321+0000","last_change":"2023-10-20T09:46:02.941197+0000","last_active":"2023-10-20T10:19:33.130321+0000","last_peered":"2023-10-20T10:19:33.130321+0000","last_clean":"2023-10-20T10:19:33.130321+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:19:33.130321+0000","last_undegraded":"2023-10-20T10:19:33.130321+0000","last_fullsized":"2023-10-20T10:19:33.130321+0000","mapping_epoch":16,"log_start":"121'441930","ondisk_log_start":"121'441930","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T19:45:00.379140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":6959030272,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":448867,"num_read_kb":1766264,"num_write":448417,"num_write_kb":1793668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"116'34","reported_seq":269,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-20T10:18:37.869240+0000","last_change":"2023-10-20T09:13:38.741487+0000","last_active":"2023-10-20T10:18:37.869240+0000","last_peered":"2023-10-20T10:18:37.869240+0000","last_clean":"2023-10-20T10:18:37.869240+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:18:37.869240+0000","last_undegraded":"2023-10-20T10:18:37.869240+0000","last_fullsized":"2023-10-20T10:18:37.869240+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T15:48:03.587791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"122'419113","reported_seq":832320,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:33.716764+0000","last_change":"2023-10-20T09:46:02.833709+0000","last_active":"2023-10-20T10:19:33.716764+0000","last_peered":"2023-10-20T10:19:33.716764+0000","last_clean":"2023-10-20T10:19:33.716764+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:19:33.716764+0000","last_undegraded":"2023-10-20T10:19:33.716764+0000","last_fullsized":"2023-10-20T10:19:33.716764+0000","mapping_epoch":16,"log_start":"121'409039","ondisk_log_start":"121'409039","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T20:34:58.352460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6451920896,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":416290,"num_read_kb":1635984,"num_write":415820,"num_write_kb":1663280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"120'309","reported_seq":6148,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:30.320070+0000","last_change":"2023-10-20T09:13:38.740624+0000","last_active":"2023-10-20T10:19:30.320070+0000","last_peered":"2023-10-20T10:19:30.320070+0000","last_clean":"2023-10-20T10:19:30.320070+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:19:30.320070+0000","last_undegraded":"2023-10-20T10:19:30.320070+0000","last_fullsized":"2023-10-20T10:19:30.320070+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:21:43.433612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"122'445099","reported_seq":884412,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:33.121072+0000","last_change":"2023-10-20T09:46:02.852528+0000","last_active":"2023-10-20T10:19:33.121072+0000","last_peered":"2023-10-20T10:19:33.121072+0000","last_clean":"2023-10-20T10:19:33.121072+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:19:33.121072+0000","last_undegraded":"2023-10-20T10:19:33.121072+0000","last_fullsized":"2023-10-20T10:19:33.121072+0000","mapping_epoch":16,"log_start":"121'435034","ondisk_log_start":"121'435034","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:22:55.065960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":6852706304,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":442651,"num_read_kb":1740136,"num_write":441562,"num_write_kb":1766248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"120'288","reported_seq":5999,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:30.330960+0000","last_change":"2023-10-20T09:13:38.738881+0000","last_active":"2023-10-20T10:19:30.330960+0000","last_peered":"2023-10-20T10:19:30.330960+0000","last_clean":"2023-10-20T10:19:30.330960+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:19:30.330960+0000","last_undegraded":"2023-10-20T10:19:30.330960+0000","last_fullsized":"2023-10-20T10:19:30.330960+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:59:24.678918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"122'443544","reported_seq":881435,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:33.711766+0000","last_change":"2023-10-20T09:46:03.048518+0000","last_active":"2023-10-20T10:19:33.711766+0000","last_peered":"2023-10-20T10:19:33.711766+0000","last_clean":"2023-10-20T10:19:33.711766+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:19:33.711766+0000","last_undegraded":"2023-10-20T10:19:33.711766+0000","last_fullsized":"2023-10-20T10:19:33.711766+0000","mapping_epoch":16,"log_start":"121'433524","ondisk_log_start":"121'433524","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:07:14.831718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6818660371,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":441156,"num_read_kb":1734476,"num_write":440070,"num_write_kb":1760274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"116'13","reported_seq":176,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-20T10:18:38.557492+0000","last_change":"2023-10-20T09:13:38.738881+0000","last_active":"2023-10-20T10:18:38.557492+0000","last_peered":"2023-10-20T10:18:38.557492+0000","last_clean":"2023-10-20T10:18:38.557492+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:18:38.557492+0000","last_undegraded":"2023-10-20T10:18:38.557492+0000","last_fullsized":"2023-10-20T10:18:38.557492+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:33:03.535449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":280,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-20T10:17:58.370519+0000","last_change":"2023-10-20T09:13:35.016203+0000","last_active":"2023-10-20T10:17:58.370519+0000","last_peered":"2023-10-20T10:17:58.370519+0000","last_clean":"2023-10-20T10:17:58.370519+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:17:58.370519+0000","last_undegraded":"2023-10-20T10:17:58.370519+0000","last_fullsized":"2023-10-20T10:17:58.370519+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:33.563853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:33.563853+0000","last_clean_scrub_stamp":"2023-10-20T09:13:33.563853+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:44:23.181197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27082317843,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1748964,"num_read_kb":6876860,"num_write":1745869,"num_write_kb":6983470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14301831168,"data_stored":27684585510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40166,"ondisk_log_size":40166,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":28472,"ondisk_log_size":28472,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739163,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8485072,"kb_used_data":6983880,"kb_used_omap":1430,"kb_used_meta":1499753,"kb_avail":85886768,"statfs":{"total":96636764160,"available":87948050432,"internally_reserved":0,"allocated":7151493120,"data_stored":13842054703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1465289,"internal_metadata":1535747127},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:18:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":0.60699999999999998,"15min":0.45000000000000001},"min":{"1min":0.16700000000000001,"5min":0.16300000000000001,"15min":0.14699999999999999},"max":{"1min":6.8419999999999996,"5min":8.2449999999999992,"15min":8.2449999999999992},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.55000000000000004,"15min":0.48299999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.8239999999999998,"5min":6.8239999999999998,"15min":6.8239999999999998},"last":0.215}]},{"osd":1,"last update":"Fri Oct 20 10:18:57 2023","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":0.72699999999999998,"15min":0.71899999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.187},"max":{"1min":8.282,"5min":8.282,"15min":35.670000000000002},"last":0.248},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.47699999999999998,"15min":0.63200000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":3.6619999999999999,"5min":4.0629999999999997,"15min":34.948999999999998},"last":0.47599999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739163,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4567848,"kb_used_data":3425248,"kb_used_omap":676,"kb_used_meta":1141915,"kb_avail":89803992,"statfs":{"total":96636764160,"available":91959287808,"internally_reserved":0,"allocated":3507453952,"data_stored":6771677603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1169321547},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.47699999999999998,"15min":0.443},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.157},"max":{"1min":4.0209999999999999,"5min":4.0209999999999999,"15min":4.7039999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.27700000000000002,"5min":0.45200000000000001,"15min":0.437},"min":{"1min":0.20000000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.376,"5min":5.8479999999999999,"15min":7.7510000000000003},"last":0.22}]},{"osd":2,"last update":"Fri Oct 20 10:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.40300000000000002,"15min":0.41799999999999998},"min":{"1min":0.20799999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.60699999999999998,"5min":3.657,"15min":4.8920000000000003},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.45100000000000001,"15min":0.42599999999999999},"min":{"1min":0.26000000000000001,"5min":0.23799999999999999,"15min":0.215},"max":{"1min":0.64000000000000001,"5min":3.6869999999999998,"15min":4.9279999999999999},"last":0.372}]}]},{"osd":0,"up_from":8,"seq":34359739163,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5093068,"kb_used_data":3561668,"kb_used_omap":698,"kb_used_meta":1530693,"kb_avail":89278772,"statfs":{"total":96636764160,"available":91421462528,"internally_reserved":0,"allocated":3647148032,"data_stored":7072350996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":1567430218},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,5,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:18:38 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":1.1799999999999999,"15min":0.72399999999999998},"min":{"1min":0.188,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":1.8240000000000001,"5min":81.197000000000003,"15min":81.197000000000003},"last":0.224},{"interface":"front","average":{"1min":0.27400000000000002,"5min":1.206,"15min":0.67200000000000004},"min":{"1min":0.20399999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.35499999999999998,"5min":80.334999999999994,"15min":80.334999999999994},"last":0.26100000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:18:38 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.61199999999999999,"15min":0.50900000000000001},"min":{"1min":0.23899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":11.176,"5min":11.176,"15min":11.176},"last":0.31},{"interface":"front","average":{"1min":1.1950000000000001,"5min":0.70999999999999996,"15min":0.624},"min":{"1min":0.26900000000000002,"5min":0.20000000000000001,"15min":0.187},"max":{"1min":11.093,"5min":11.093,"15min":11.093},"last":0.29199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3645353984,"data_stored":7071457280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3505733632,"data_stored":6771007507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7150743552,"data_stored":13842120723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:19:37.343 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-20T10:19:37.343 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:19:37.997 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:19:37.999 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:19:38.041 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2103,"stamp":"2023-10-20T10:19:37.201832+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27082912259,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1787635,"num_read_kb":6909581,"num_write":1746883,"num_write_kb":6985783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68735,"ondisk_log_size":68735,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18145988,"kb_used_data":13970796,"kb_used_omap":2805,"kb_used_meta":4172362,"kb_avail":264969532,"statfs":{"total":289910292480,"available":271328800768,"internally_reserved":0,"allocated":14306095104,"data_stored":27686083302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2873140,"internal_metadata":4272498892},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,10,7,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17686528,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4236,"num_read_kb":16892,"num_write":4292,"num_write_kb":17168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002083"},"pg_stats":[{"pgid":"2.7","version":"122'8915","reported_seq":13400,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:32.985139+0000","last_change":"2023-10-20T09:13:38.742871+0000","last_active":"2023-10-20T10:19:32.985139+0000","last_peered":"2023-10-20T10:19:32.985139+0000","last_clean":"2023-10-20T10:19:32.985139+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:19:32.985139+0000","last_undegraded":"2023-10-20T10:19:32.985139+0000","last_fullsized":"2023-10-20T10:19:32.985139+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":8915,"log_dups_size":0,"ondisk_log_size":8915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:18:55.451570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"122'17523","reported_seq":27498,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:33.009850+0000","last_change":"2023-10-20T09:13:38.742974+0000","last_active":"2023-10-20T10:19:33.009850+0000","last_peered":"2023-10-20T10:19:33.009850+0000","last_clean":"2023-10-20T10:19:33.009850+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:19:33.009850+0000","last_undegraded":"2023-10-20T10:19:33.009850+0000","last_fullsized":"2023-10-20T10:19:33.009850+0000","mapping_epoch":13,"log_start":"92'7500","ondisk_log_start":"92'7500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:38:55.327279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"101'10","reported_seq":180,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-20T10:18:38.557537+0000","last_change":"2023-10-20T09:13:38.742635+0000","last_active":"2023-10-20T10:18:38.557537+0000","last_peered":"2023-10-20T10:18:38.557537+0000","last_clean":"2023-10-20T10:18:38.557537+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:18:38.557537+0000","last_undegraded":"2023-10-20T10:18:38.557537+0000","last_fullsized":"2023-10-20T10:18:38.557537+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:42:15.061080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"122'8880","reported_seq":14139,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:32.985180+0000","last_change":"2023-10-20T09:13:38.742777+0000","last_active":"2023-10-20T10:19:32.985180+0000","last_peered":"2023-10-20T10:19:32.985180+0000","last_clean":"2023-10-20T10:19:32.985180+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:19:32.985180+0000","last_undegraded":"2023-10-20T10:19:32.985180+0000","last_fullsized":"2023-10-20T10:19:32.985180+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":8880,"log_dups_size":0,"ondisk_log_size":8880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:36:13.287316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"122'451937","reported_seq":897491,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:33.130321+0000","last_change":"2023-10-20T09:46:02.941197+0000","last_active":"2023-10-20T10:19:33.130321+0000","last_peered":"2023-10-20T10:19:33.130321+0000","last_clean":"2023-10-20T10:19:33.130321+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:19:33.130321+0000","last_undegraded":"2023-10-20T10:19:33.130321+0000","last_fullsized":"2023-10-20T10:19:33.130321+0000","mapping_epoch":16,"log_start":"121'441930","ondisk_log_start":"121'441930","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T19:45:00.379140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":6959030272,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":448867,"num_read_kb":1766264,"num_write":448417,"num_write_kb":1793668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"116'34","reported_seq":269,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-20T10:18:37.869240+0000","last_change":"2023-10-20T09:13:38.741487+0000","last_active":"2023-10-20T10:18:37.869240+0000","last_peered":"2023-10-20T10:18:37.869240+0000","last_clean":"2023-10-20T10:18:37.869240+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:18:37.869240+0000","last_undegraded":"2023-10-20T10:18:37.869240+0000","last_fullsized":"2023-10-20T10:18:37.869240+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T15:48:03.587791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"122'419113","reported_seq":832320,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:33.716764+0000","last_change":"2023-10-20T09:46:02.833709+0000","last_active":"2023-10-20T10:19:33.716764+0000","last_peered":"2023-10-20T10:19:33.716764+0000","last_clean":"2023-10-20T10:19:33.716764+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:19:33.716764+0000","last_undegraded":"2023-10-20T10:19:33.716764+0000","last_fullsized":"2023-10-20T10:19:33.716764+0000","mapping_epoch":16,"log_start":"121'409039","ondisk_log_start":"121'409039","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T20:34:58.352460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6451920896,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":416290,"num_read_kb":1635984,"num_write":415820,"num_write_kb":1663280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"120'309","reported_seq":6148,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:30.320070+0000","last_change":"2023-10-20T09:13:38.740624+0000","last_active":"2023-10-20T10:19:30.320070+0000","last_peered":"2023-10-20T10:19:30.320070+0000","last_clean":"2023-10-20T10:19:30.320070+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:19:30.320070+0000","last_undegraded":"2023-10-20T10:19:30.320070+0000","last_fullsized":"2023-10-20T10:19:30.320070+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:21:43.433612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"122'445099","reported_seq":884412,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:33.121072+0000","last_change":"2023-10-20T09:46:02.852528+0000","last_active":"2023-10-20T10:19:33.121072+0000","last_peered":"2023-10-20T10:19:33.121072+0000","last_clean":"2023-10-20T10:19:33.121072+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:19:33.121072+0000","last_undegraded":"2023-10-20T10:19:33.121072+0000","last_fullsized":"2023-10-20T10:19:33.121072+0000","mapping_epoch":16,"log_start":"121'435034","ondisk_log_start":"121'435034","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:22:55.065960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":6852706304,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":442651,"num_read_kb":1740136,"num_write":441562,"num_write_kb":1766248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"120'288","reported_seq":5999,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:30.330960+0000","last_change":"2023-10-20T09:13:38.738881+0000","last_active":"2023-10-20T10:19:30.330960+0000","last_peered":"2023-10-20T10:19:30.330960+0000","last_clean":"2023-10-20T10:19:30.330960+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:19:30.330960+0000","last_undegraded":"2023-10-20T10:19:30.330960+0000","last_fullsized":"2023-10-20T10:19:30.330960+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:59:24.678918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"122'443544","reported_seq":881435,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:33.711766+0000","last_change":"2023-10-20T09:46:03.048518+0000","last_active":"2023-10-20T10:19:33.711766+0000","last_peered":"2023-10-20T10:19:33.711766+0000","last_clean":"2023-10-20T10:19:33.711766+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:19:33.711766+0000","last_undegraded":"2023-10-20T10:19:33.711766+0000","last_fullsized":"2023-10-20T10:19:33.711766+0000","mapping_epoch":16,"log_start":"121'433524","ondisk_log_start":"121'433524","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:07:14.831718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6818660371,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":441156,"num_read_kb":1734476,"num_write":440070,"num_write_kb":1760274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"116'13","reported_seq":176,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-20T10:18:38.557492+0000","last_change":"2023-10-20T09:13:38.738881+0000","last_active":"2023-10-20T10:18:38.557492+0000","last_peered":"2023-10-20T10:18:38.557492+0000","last_clean":"2023-10-20T10:18:38.557492+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:18:38.557492+0000","last_undegraded":"2023-10-20T10:18:38.557492+0000","last_fullsized":"2023-10-20T10:18:38.557492+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:33:03.535449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":280,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-20T10:17:58.370519+0000","last_change":"2023-10-20T09:13:35.016203+0000","last_active":"2023-10-20T10:17:58.370519+0000","last_peered":"2023-10-20T10:17:58.370519+0000","last_clean":"2023-10-20T10:17:58.370519+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:17:58.370519+0000","last_undegraded":"2023-10-20T10:17:58.370519+0000","last_fullsized":"2023-10-20T10:17:58.370519+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:33.563853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:33.563853+0000","last_clean_scrub_stamp":"2023-10-20T09:13:33.563853+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:44:23.181197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27082317843,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1748964,"num_read_kb":6876860,"num_write":1745869,"num_write_kb":6983470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14301831168,"data_stored":27684585510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40166,"ondisk_log_size":40166,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":28472,"ondisk_log_size":28472,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739163,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8485072,"kb_used_data":6983880,"kb_used_omap":1430,"kb_used_meta":1499753,"kb_avail":85886768,"statfs":{"total":96636764160,"available":87948050432,"internally_reserved":0,"allocated":7151493120,"data_stored":13842054703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1465289,"internal_metadata":1535747127},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:18:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":0.60699999999999998,"15min":0.45000000000000001},"min":{"1min":0.16700000000000001,"5min":0.16300000000000001,"15min":0.14699999999999999},"max":{"1min":6.8419999999999996,"5min":8.2449999999999992,"15min":8.2449999999999992},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.55000000000000004,"15min":0.48299999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.8239999999999998,"5min":6.8239999999999998,"15min":6.8239999999999998},"last":0.215}]},{"osd":1,"last update":"Fri Oct 20 10:18:57 2023","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":0.72699999999999998,"15min":0.71899999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.187},"max":{"1min":8.282,"5min":8.282,"15min":35.670000000000002},"last":0.248},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.47699999999999998,"15min":0.63200000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":3.6619999999999999,"5min":4.0629999999999997,"15min":34.948999999999998},"last":0.47599999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739163,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4567848,"kb_used_data":3425248,"kb_used_omap":676,"kb_used_meta":1141915,"kb_avail":89803992,"statfs":{"total":96636764160,"available":91959287808,"internally_reserved":0,"allocated":3507453952,"data_stored":6771677603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1169321547},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.47699999999999998,"15min":0.443},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.157},"max":{"1min":4.0209999999999999,"5min":4.0209999999999999,"15min":4.7039999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.27700000000000002,"5min":0.45200000000000001,"15min":0.437},"min":{"1min":0.20000000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.376,"5min":5.8479999999999999,"15min":7.7510000000000003},"last":0.22}]},{"osd":2,"last update":"Fri Oct 20 10:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.40300000000000002,"15min":0.41799999999999998},"min":{"1min":0.20799999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.60699999999999998,"5min":3.657,"15min":4.8920000000000003},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.45100000000000001,"15min":0.42599999999999999},"min":{"1min":0.26000000000000001,"5min":0.23799999999999999,"15min":0.215},"max":{"1min":0.64000000000000001,"5min":3.6869999999999998,"15min":4.9279999999999999},"last":0.372}]}]},{"osd":0,"up_from":8,"seq":34359739163,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5093068,"kb_used_data":3561668,"kb_used_omap":698,"kb_used_meta":1530693,"kb_avail":89278772,"statfs":{"total":96636764160,"available":91421462528,"internally_reserved":0,"allocated":3647148032,"data_stored":7072350996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":1567430218},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,5,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:18:38 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":1.1799999999999999,"15min":0.72399999999999998},"min":{"1min":0.188,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":1.8240000000000001,"5min":81.197000000000003,"15min":81.197000000000003},"last":0.224},{"interface":"front","average":{"1min":0.27400000000000002,"5min":1.206,"15min":0.67200000000000004},"min":{"1min":0.20399999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.35499999999999998,"5min":80.334999999999994,"15min":80.334999999999994},"last":0.26100000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:18:38 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.61199999999999999,"15min":0.50900000000000001},"min":{"1min":0.23899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":11.176,"5min":11.176,"15min":11.176},"last":0.31},{"interface":"front","average":{"1min":1.1950000000000001,"5min":0.70999999999999996,"15min":0.624},"min":{"1min":0.26900000000000002,"5min":0.20000000000000001,"15min":0.187},"max":{"1min":11.093,"5min":11.093,"15min":11.093},"last":0.29199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3645353984,"data_stored":7071457280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3505733632,"data_stored":6771007507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7150743552,"data_stored":13842120723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:19:38.043 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T10:19:38.595 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:19:38.595 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":122,"fsid":"ab6e8fb1-5f4f-4d44-a016-fff308aa08d4","created":"2023-10-20T09:13:21.742700+0000","modified":"2023-10-20T10:19:02.987824+0000","last_up_change":"2023-10-20T09:13:31.560690+0000","last_in_change":"2023-10-20T09:13:22.618839+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T09:13:32.620929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T09:13:37.030861+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2023-10-20T09:13:42.826174+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":89,"snap_epoch":116,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"234f6989-5b92-4cb0-b780-06ab9137710a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":3924958301},{"type":"v1","addr":"172.21.15.150:6807","nonce":3924958301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":3924958301},{"type":"v1","addr":"172.21.15.150:6811","nonce":3924958301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3924958301},{"type":"v1","addr":"172.21.15.150:6815","nonce":3924958301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3924958301},{"type":"v1","addr":"172.21.15.150:6813","nonce":3924958301}]},"public_addr":"172.21.15.150:6807/3924958301","cluster_addr":"172.21.15.150:6811/3924958301","heartbeat_back_addr":"172.21.15.150:6815/3924958301","heartbeat_front_addr":"172.21.15.150:6813/3924958301","state":["exists","up"]},{"osd":1,"uuid":"bb6392bc-9616-4abc-95b2-83d9879ad204","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1576706561},{"type":"v1","addr":"172.21.15.150:6801","nonce":1576706561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1576706561},{"type":"v1","addr":"172.21.15.150:6803","nonce":1576706561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1576706561},{"type":"v1","addr":"172.21.15.150:6810","nonce":1576706561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6805","nonce":1576706561},{"type":"v1","addr":"172.21.15.150:6806","nonce":1576706561}]},"public_addr":"172.21.15.150:6801/1576706561","cluster_addr":"172.21.15.150:6803/1576706561","heartbeat_back_addr":"172.21.15.150:6810/1576706561","heartbeat_front_addr":"172.21.15.150:6806/1576706561","state":["exists","up"]},{"osd":2,"uuid":"38363eb3-42d5-4a63-b24a-50666cca82c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2945223909},{"type":"v1","addr":"172.21.15.150:6817","nonce":2945223909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":2945223909},{"type":"v1","addr":"172.21.15.150:6819","nonce":2945223909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":2945223909},{"type":"v1","addr":"172.21.15.150:6823","nonce":2945223909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2945223909},{"type":"v1","addr":"172.21.15.150:6821","nonce":2945223909}]},"public_addr":"172.21.15.150:6817/2945223909","cluster_addr":"172.21.15.150:6819/2945223909","heartbeat_back_addr":"172.21.15.150:6823/2945223909","heartbeat_front_addr":"172.21.15.150:6821/2945223909","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T09:13:28.131113+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T09:13:27.588549+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T09:13:27.784784+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:0/4059769879":"2023-10-20T10:51:54.952148+0000","172.21.15.150:0/3966202163":"2023-10-20T10:50:46.726486+0000","172.21.15.150:0/1658214644":"2023-10-20T10:49:26.995087+0000","172.21.15.150:0/1670666381":"2023-10-20T10:26:10.609306+0000","172.21.15.150:0/1998324367":"2023-10-20T10:21:26.552290+0000","172.21.15.150:0/935800402":"2023-10-20T10:20:18.625984+0000","172.21.15.150:0/1398764105":"2023-10-20T10:34:23.057779+0000","172.21.15.150:0/569130003":"2023-10-20T10:27:18.987352+0000","172.21.15.150:0/998694798":"2023-10-20T10:38:58.236658+0000","172.21.15.150:0/587169855":"2023-10-20T10:22:34.962995+0000","172.21.15.150:0/4240815768":"2023-10-20T10:43:44.758650+0000","172.21.15.150:0/2805376037":"2023-10-20T10:28:27.756742+0000","172.21.15.150:0/2500377447":"2023-10-20T10:33:14.367391+0000","172.21.15.150:0/2268406928":"2023-10-20T10:25:02.859431+0000","172.21.15.150:0/1510323101":"2023-10-20T10:40:06.859675+0000","172.21.15.150:0/171431162":"2023-10-20T10:48:29.343892+0000","172.21.15.150:0/22457227":"2023-10-20T10:23:42.948955+0000","172.21.15.150:0/1360796600":"2023-10-20T10:30:56.310558+0000","172.21.15.150:0/840525808":"2023-10-20T10:32:05.660798+0000","172.21.15.150:0/260822388":"2023-10-20T10:29:36.133241+0000","172.21.15.150:0/3786067858":"2023-10-20T10:35:32.280849+0000","172.21.15.150:0/348031646":"2023-10-20T10:36:40.706595+0000","172.21.15.150:0/389382636":"2023-10-20T10:46:01.634071+0000","172.21.15.150:0/3774459303":"2023-10-20T10:53:03.312401+0000","172.21.15.150:0/2624331567":"2023-10-20T10:41:16.202134+0000","172.21.15.150:0/2299851903":"2023-10-20T10:37:49.781906+0000","172.21.15.150:0/249834779":"2023-10-20T10:47:09.715414+0000","172.21.15.150:0/3221254932":"2023-10-20T10:42:36.059853+0000","172.21.15.150:0/436164353":"2023-10-20T10:44:53.038557+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T10:19:39.622 INFO:tasks.ceph:Scrubbing osd.0 2023-10-20T10:19:39.622 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-20T10:19:39.887 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2023-10-20T10:19:39.888 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-20T10:19:39.888 INFO:teuthology.orchestra.run.smithi150.stdout:} 2023-10-20T10:19:39.924 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-20T10:19:40.431 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 0 to deep-scrub 2023-10-20T10:19:40.476 INFO:tasks.ceph:Scrubbing osd.1 2023-10-20T10:19:40.476 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-20T10:19:40.738 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2023-10-20T10:19:40.739 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-20T10:19:40.739 INFO:teuthology.orchestra.run.smithi150.stdout:} 2023-10-20T10:19:40.778 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-20T10:19:41.247 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 1 to deep-scrub 2023-10-20T10:19:41.284 INFO:tasks.ceph:Scrubbing osd.2 2023-10-20T10:19:41.285 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-20T10:19:41.517 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2023-10-20T10:19:41.517 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-20T10:19:41.518 INFO:teuthology.orchestra.run.smithi150.stdout:} 2023-10-20T10:19:41.556 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-20T10:19:42.139 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 2 to deep-scrub 2023-10-20T10:19:42.178 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:19:42.709 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:19:42.710 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:19:42.737 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2105,"stamp":"2023-10-20T10:19:41.202495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27091370499,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1789654,"num_read_kb":6917629,"num_write":1748924,"num_write_kb":6993947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68824,"ondisk_log_size":68824,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18207924,"kb_used_data":13986972,"kb_used_omap":2805,"kb_used_meta":4218122,"kb_avail":264907596,"statfs":{"total":289910292480,"available":271265378304,"internally_reserved":0,"allocated":14322659328,"data_stored":27702647526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2873140,"internal_metadata":4319357132},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,7,6,6,14,14,15,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17264640,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4198,"num_read_kb":16740,"num_write":4180,"num_write_kb":16720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002103"},"pg_stats":[{"pgid":"2.7","version":"122'8927","reported_seq":13415,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:38.149049+0000","last_change":"2023-10-20T09:13:38.742871+0000","last_active":"2023-10-20T10:19:38.149049+0000","last_peered":"2023-10-20T10:19:38.149049+0000","last_clean":"2023-10-20T10:19:38.149049+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:19:38.149049+0000","last_undegraded":"2023-10-20T10:19:38.149049+0000","last_fullsized":"2023-10-20T10:19:38.149049+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":8927,"log_dups_size":0,"ondisk_log_size":8927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:18:55.451570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"122'17547","reported_seq":27528,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:38.176832+0000","last_change":"2023-10-20T09:13:38.742974+0000","last_active":"2023-10-20T10:19:38.176832+0000","last_peered":"2023-10-20T10:19:38.176832+0000","last_clean":"2023-10-20T10:19:38.176832+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:19:38.176832+0000","last_undegraded":"2023-10-20T10:19:38.176832+0000","last_fullsized":"2023-10-20T10:19:38.176832+0000","mapping_epoch":13,"log_start":"92'7500","ondisk_log_start":"92'7500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:38:55.327279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"101'10","reported_seq":180,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-20T10:18:38.557537+0000","last_change":"2023-10-20T09:13:38.742635+0000","last_active":"2023-10-20T10:18:38.557537+0000","last_peered":"2023-10-20T10:18:38.557537+0000","last_clean":"2023-10-20T10:18:38.557537+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:18:38.557537+0000","last_undegraded":"2023-10-20T10:18:38.557537+0000","last_fullsized":"2023-10-20T10:18:38.557537+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:42:15.061080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"122'8892","reported_seq":14154,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:38.149744+0000","last_change":"2023-10-20T09:13:38.742777+0000","last_active":"2023-10-20T10:19:38.149744+0000","last_peered":"2023-10-20T10:19:38.149744+0000","last_clean":"2023-10-20T10:19:38.149744+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:19:38.149744+0000","last_undegraded":"2023-10-20T10:19:38.149744+0000","last_fullsized":"2023-10-20T10:19:38.149744+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":8892,"log_dups_size":0,"ondisk_log_size":8892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:36:13.287316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"122'452473","reported_seq":898542,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:38.182712+0000","last_change":"2023-10-20T09:46:02.941197+0000","last_active":"2023-10-20T10:19:38.182712+0000","last_peered":"2023-10-20T10:19:38.182712+0000","last_clean":"2023-10-20T10:19:38.182712+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:19:38.182712+0000","last_undegraded":"2023-10-20T10:19:38.182712+0000","last_fullsized":"2023-10-20T10:19:38.182712+0000","mapping_epoch":16,"log_start":"121'442430","ondisk_log_start":"121'442430","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T19:45:00.379140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":6961254400,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":449392,"num_read_kb":1768360,"num_write":448953,"num_write_kb":1795812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"116'34","reported_seq":269,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-20T10:18:37.869240+0000","last_change":"2023-10-20T09:13:38.741487+0000","last_active":"2023-10-20T10:18:37.869240+0000","last_peered":"2023-10-20T10:18:37.869240+0000","last_clean":"2023-10-20T10:18:37.869240+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:18:37.869240+0000","last_undegraded":"2023-10-20T10:18:37.869240+0000","last_fullsized":"2023-10-20T10:18:37.869240+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T15:48:03.587791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"122'419599","reported_seq":833279,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:38.738194+0000","last_change":"2023-10-20T09:46:02.833709+0000","last_active":"2023-10-20T10:19:38.738194+0000","last_peered":"2023-10-20T10:19:38.738194+0000","last_clean":"2023-10-20T10:19:38.738194+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:19:38.738194+0000","last_undegraded":"2023-10-20T10:19:38.738194+0000","last_fullsized":"2023-10-20T10:19:38.738194+0000","mapping_epoch":16,"log_start":"121'409539","ondisk_log_start":"121'409539","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T20:34:58.352460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6453936128,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":416765,"num_read_kb":1637880,"num_write":416306,"num_write_kb":1665224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"120'309","reported_seq":6152,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:35.456253+0000","last_change":"2023-10-20T09:13:38.740624+0000","last_active":"2023-10-20T10:19:35.456253+0000","last_peered":"2023-10-20T10:19:35.456253+0000","last_clean":"2023-10-20T10:19:35.456253+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:19:35.456253+0000","last_undegraded":"2023-10-20T10:19:35.456253+0000","last_fullsized":"2023-10-20T10:19:35.456253+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:21:43.433612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"122'445606","reported_seq":885426,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:38.152986+0000","last_change":"2023-10-20T09:46:02.852528+0000","last_active":"2023-10-20T10:19:38.152986+0000","last_peered":"2023-10-20T10:19:38.152986+0000","last_clean":"2023-10-20T10:19:38.152986+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:19:38.152986+0000","last_undegraded":"2023-10-20T10:19:38.152986+0000","last_fullsized":"2023-10-20T10:19:38.152986+0000","mapping_epoch":16,"log_start":"121'435534","ondisk_log_start":"121'435534","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:22:55.065960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":6854782976,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":443153,"num_read_kb":1742136,"num_write":442069,"num_write_kb":1768276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"120'288","reported_seq":6003,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:35.478587+0000","last_change":"2023-10-20T09:13:38.738881+0000","last_active":"2023-10-20T10:19:35.478587+0000","last_peered":"2023-10-20T10:19:35.478587+0000","last_clean":"2023-10-20T10:19:35.478587+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:19:35.478587+0000","last_undegraded":"2023-10-20T10:19:35.478587+0000","last_fullsized":"2023-10-20T10:19:35.478587+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:59:24.678918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"122'444056","reported_seq":882460,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:38.746112+0000","last_change":"2023-10-20T09:46:03.048518+0000","last_active":"2023-10-20T10:19:38.746112+0000","last_peered":"2023-10-20T10:19:38.746112+0000","last_clean":"2023-10-20T10:19:38.746112+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:19:38.746112+0000","last_undegraded":"2023-10-20T10:19:38.746112+0000","last_fullsized":"2023-10-20T10:19:38.746112+0000","mapping_epoch":16,"log_start":"121'434024","ondisk_log_start":"121'434024","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:07:14.831718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6820802579,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":441673,"num_read_kb":1736532,"num_write":440582,"num_write_kb":1762322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"116'13","reported_seq":176,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-20T10:18:38.557492+0000","last_change":"2023-10-20T09:13:38.738881+0000","last_active":"2023-10-20T10:18:38.557492+0000","last_peered":"2023-10-20T10:18:38.557492+0000","last_clean":"2023-10-20T10:18:38.557492+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:18:38.557492+0000","last_undegraded":"2023-10-20T10:18:38.557492+0000","last_fullsized":"2023-10-20T10:18:38.557492+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:37.714877+0000","last_clean_scrub_stamp":"2023-10-20T09:13:37.714877+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:33:03.535449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":280,"reported_epoch":121,"state":"active+clean","last_fresh":"2023-10-20T10:17:58.370519+0000","last_change":"2023-10-20T09:13:35.016203+0000","last_active":"2023-10-20T10:17:58.370519+0000","last_peered":"2023-10-20T10:17:58.370519+0000","last_clean":"2023-10-20T10:17:58.370519+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:17:58.370519+0000","last_undegraded":"2023-10-20T10:17:58.370519+0000","last_fullsized":"2023-10-20T10:17:58.370519+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:33.563853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:33.563853+0000","last_clean_scrub_stamp":"2023-10-20T09:13:33.563853+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:44:23.181197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27090776083,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1750983,"num_read_kb":6884908,"num_write":1747910,"num_write_kb":6991634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14318469120,"data_stored":27701223462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40207,"ondisk_log_size":40207,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":28520,"ondisk_log_size":28520,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739164,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8512624,"kb_used_data":6992040,"kb_used_omap":1430,"kb_used_meta":1519145,"kb_avail":85859216,"statfs":{"total":96636764160,"available":87919837184,"internally_reserved":0,"allocated":7159848960,"data_stored":13850410543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1465289,"internal_metadata":1555604535},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:18:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":0.60699999999999998,"15min":0.45000000000000001},"min":{"1min":0.16700000000000001,"5min":0.16300000000000001,"15min":0.14699999999999999},"max":{"1min":6.8419999999999996,"5min":8.2449999999999992,"15min":8.2449999999999992},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.55000000000000004,"15min":0.48299999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.8239999999999998,"5min":6.8239999999999998,"15min":6.8239999999999998},"last":0.69899999999999995}]},{"osd":1,"last update":"Fri Oct 20 10:18:57 2023","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":0.72699999999999998,"15min":0.71899999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.187},"max":{"1min":8.282,"5min":8.282,"15min":35.670000000000002},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.47699999999999998,"15min":0.63200000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":3.6619999999999999,"5min":4.0629999999999997,"15min":34.948999999999998},"last":0.27500000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739164,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4593984,"kb_used_data":3429176,"kb_used_omap":676,"kb_used_meta":1164123,"kb_avail":89777856,"statfs":{"total":96636764160,"available":91932524544,"internally_reserved":0,"allocated":3511476224,"data_stored":6775699875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1192062539},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,3,2,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.47699999999999998,"15min":0.443},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.157},"max":{"1min":4.0209999999999999,"5min":4.0209999999999999,"15min":4.7039999999999997},"last":3.5070000000000001},{"interface":"front","average":{"1min":0.27700000000000002,"5min":0.45200000000000001,"15min":0.437},"min":{"1min":0.20000000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.376,"5min":5.8479999999999999,"15min":7.7510000000000003},"last":0.219}]},{"osd":2,"last update":"Fri Oct 20 10:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.40300000000000002,"15min":0.41799999999999998},"min":{"1min":0.20799999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.60699999999999998,"5min":3.657,"15min":4.8920000000000003},"last":0.317},{"interface":"front","average":{"1min":0.377,"5min":0.45100000000000001,"15min":0.42599999999999999},"min":{"1min":0.26000000000000001,"5min":0.23799999999999999,"15min":0.215},"max":{"1min":0.64000000000000001,"5min":3.6869999999999998,"15min":4.9279999999999999},"last":0.35299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739164,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5101316,"kb_used_data":3565756,"kb_used_omap":698,"kb_used_meta":1534853,"kb_avail":89270524,"statfs":{"total":96636764160,"available":91413016576,"internally_reserved":0,"allocated":3651334144,"data_stored":7076537108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":1571690058},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,6,1,10,8,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:18:38 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":1.1799999999999999,"15min":0.72399999999999998},"min":{"1min":0.188,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":1.8240000000000001,"5min":81.197000000000003,"15min":81.197000000000003},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.27400000000000002,"5min":1.206,"15min":0.67200000000000004},"min":{"1min":0.20399999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.35499999999999998,"5min":80.334999999999994,"15min":80.334999999999994},"last":7.2690000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:18:38 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.61199999999999999,"15min":0.50900000000000001},"min":{"1min":0.23899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":11.176,"5min":11.176,"15min":11.176},"last":0.29099999999999998},{"interface":"front","average":{"1min":1.1950000000000001,"5min":0.70999999999999996,"15min":0.624},"min":{"1min":0.26900000000000002,"5min":0.20000000000000001,"15min":0.187},"max":{"1min":11.093,"5min":11.093,"15min":11.093},"last":0.24299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3649626112,"data_stored":7075729408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3509809152,"data_stored":6775083027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7159033856,"data_stored":13850411027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:19:42.741 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-20T09:13:37.714877+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=37, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:19:42.741 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-20T09:13:37.714877+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=37, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:19:42.742 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-20T09:13:37.714877+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=37, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:19:42.742 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-20T09:13:37.714877+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=37, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:19:42.742 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:19:42.743 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-20T09:13:37.714877+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=37, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:19:42.743 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:19:42.743 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-20T09:13:37.714877+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=37, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:19:42.744 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:19:42.744 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-20T09:13:37.714877+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=37, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:19:42.745 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:19:42.745 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-20T09:13:37.714877+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=37, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:19:42.746 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-20T09:13:33.563853+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=33, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:19:42.746 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:19:43.243 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:19:43.242+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:19:43.256 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:19:43.254+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:20:02.747 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:20:03.346 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:20:03.347 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:20:03.400 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2116,"stamp":"2023-10-20T10:20:03.206313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27123360259,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1797298,"num_read_kb":6948053,"num_write":1756720,"num_write_kb":7025131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68948,"ondisk_log_size":68948,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18410088,"kb_used_data":14048976,"kb_used_omap":2805,"kb_used_meta":4358282,"kb_avail":264705432,"statfs":{"total":289910292480,"available":271058362368,"internally_reserved":0,"allocated":14386151424,"data_stored":27766139622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2873140,"internal_metadata":4462880972},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,3,5,15,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15835136,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3806,"num_read_kb":15140,"num_write":3868,"num_write_kb":15472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6458,"num_omap_keys":119,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002097"},"pg_stats":[{"pgid":"2.7","version":"122'8971","reported_seq":13480,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:58.808676+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:19:58.808676+0000","last_peered":"2023-10-20T10:19:58.808676+0000","last_clean":"2023-10-20T10:19:58.808676+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:19:58.808676+0000","last_undegraded":"2023-10-20T10:19:58.808676+0000","last_fullsized":"2023-10-20T10:19:58.808676+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":8971,"log_dups_size":0,"ondisk_log_size":8971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"122'17621","reported_seq":27635,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:58.849157+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:19:58.849157+0000","last_peered":"2023-10-20T10:19:58.849157+0000","last_clean":"2023-10-20T10:19:58.849157+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:19:58.849157+0000","last_undegraded":"2023-10-20T10:19:58.849157+0000","last_fullsized":"2023-10-20T10:19:58.849157+0000","mapping_epoch":13,"log_start":"92'7600","ondisk_log_start":"92'7600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":199,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:55.096661+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:19:55.096661+0000","last_peered":"2023-10-20T10:19:55.096661+0000","last_clean":"2023-10-20T10:19:55.096661+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:19:55.096661+0000","last_undegraded":"2023-10-20T10:19:55.096661+0000","last_fullsized":"2023-10-20T10:19:55.096661+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"122'8934","reported_seq":14217,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:58.819234+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:19:58.819234+0000","last_peered":"2023-10-20T10:19:58.819234+0000","last_clean":"2023-10-20T10:19:58.819234+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:19:58.819234+0000","last_undegraded":"2023-10-20T10:19:58.819234+0000","last_fullsized":"2023-10-20T10:19:58.819234+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":8934,"log_dups_size":0,"ondisk_log_size":8934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"122'454484","reported_seq":902511,"reported_epoch":122,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:19:58.269147+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:19:58.269147+0000","last_peered":"2023-10-20T10:19:58.269147+0000","last_clean":"2023-10-20T10:19:58.269147+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:19:58.269147+0000","last_undegraded":"2023-10-20T10:19:58.269147+0000","last_fullsized":"2023-10-20T10:19:58.269147+0000","mapping_epoch":16,"log_start":"121'444430","ondisk_log_start":"121'444430","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":64,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 14s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":6969286656,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":451328,"num_read_kb":1776080,"num_write":450918,"num_write_kb":1803672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":293,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:41.970652+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:19:41.970652+0000","last_peered":"2023-10-20T10:19:41.970652+0000","last_clean":"2023-10-20T10:19:41.970652+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:19:41.970652+0000","last_undegraded":"2023-10-20T10:19:41.970652+0000","last_fullsized":"2023-10-20T10:19:41.970652+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"122'421417","reported_seq":836937,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:58.821770+0000","last_change":"2023-10-20T09:46:02.833709+0000","last_active":"2023-10-20T10:19:58.821770+0000","last_peered":"2023-10-20T10:19:58.821770+0000","last_clean":"2023-10-20T10:19:58.821770+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:19:58.821770+0000","last_undegraded":"2023-10-20T10:19:58.821770+0000","last_fullsized":"2023-10-20T10:19:58.821770+0000","mapping_epoch":16,"log_start":"121'411339","ondisk_log_start":"121'411339","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6461378560,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":418604,"num_read_kb":1645188,"num_write":418124,"num_write_kb":1672496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"122'326","reported_seq":6194,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:55.925937+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:19:55.925937+0000","last_peered":"2023-10-20T10:19:55.925937+0000","last_clean":"2023-10-20T10:19:55.925937+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:19:55.925937+0000","last_undegraded":"2023-10-20T10:19:55.925937+0000","last_fullsized":"2023-10-20T10:19:55.925937+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"122'447673","reported_seq":889453,"reported_epoch":122,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:19:58.273596+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:19:58.273596+0000","last_peered":"2023-10-20T10:19:58.273596+0000","last_clean":"2023-10-20T10:19:58.273596+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:19:58.273596+0000","last_undegraded":"2023-10-20T10:19:58.273596+0000","last_fullsized":"2023-10-20T10:19:58.273596+0000","mapping_epoch":16,"log_start":"121'437634","ondisk_log_start":"121'437634","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":66,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 13s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":6863028224,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":445098,"num_read_kb":1749868,"num_write":444086,"num_write_kb":1776344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"122'303","reported_seq":6043,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:55.922664+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:19:55.922664+0000","last_peered":"2023-10-20T10:19:55.922664+0000","last_clean":"2023-10-20T10:19:55.922664+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:19:55.922664+0000","last_undegraded":"2023-10-20T10:19:55.922664+0000","last_fullsized":"2023-10-20T10:19:55.922664+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"122'446052","reported_seq":886382,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:58.816081+0000","last_change":"2023-10-20T09:46:03.048518+0000","last_active":"2023-10-20T10:19:58.816081+0000","last_peered":"2023-10-20T10:19:58.816081+0000","last_clean":"2023-10-20T10:19:58.816081+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:19:58.816081+0000","last_undegraded":"2023-10-20T10:19:58.816081+0000","last_fullsized":"2023-10-20T10:19:58.816081+0000","mapping_epoch":16,"log_start":"121'436024","ondisk_log_start":"121'436024","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6829072403,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":443597,"num_read_kb":1744196,"num_write":442578,"num_write_kb":1770306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":198,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:51.179153+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:19:51.179153+0000","last_peered":"2023-10-20T10:19:51.179153+0000","last_clean":"2023-10-20T10:19:51.179153+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:19:51.179153+0000","last_undegraded":"2023-10-20T10:19:51.179153+0000","last_fullsized":"2023-10-20T10:19:51.179153+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":292,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:42.469749+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:19:42.469749+0000","last_peered":"2023-10-20T10:19:42.469749+0000","last_clean":"2023-10-20T10:19:42.469749+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:19:42.469749+0000","last_undegraded":"2023-10-20T10:19:42.469749+0000","last_fullsized":"2023-10-20T10:19:42.469749+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27122765843,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1758627,"num_read_kb":6915332,"num_write":1755706,"num_write_kb":7022818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14382473216,"data_stored":27765227558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40199,"ondisk_log_size":40199,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":28650,"ondisk_log_size":28650,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739168,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8611668,"kb_used_data":7023052,"kb_used_omap":1430,"kb_used_meta":1587177,"kb_avail":85760172,"statfs":{"total":96636764160,"available":87818416128,"internally_reserved":0,"allocated":7191605248,"data_stored":13882166831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1465289,"internal_metadata":1625269303},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:19:58 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.626,"15min":0.45100000000000001},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.14699999999999999},"max":{"1min":3.9870000000000001,"5min":8.2449999999999992,"15min":8.2449999999999992},"last":0.245},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57699999999999996,"15min":0.501},"min":{"1min":0.215,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.9749999999999996,"5min":6.8239999999999998,"15min":6.8239999999999998},"last":4.9749999999999996}]},{"osd":1,"last update":"Fri Oct 20 10:19:58 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.746,"15min":0.71899999999999997},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":4.8659999999999997,"5min":8.282,"15min":35.670000000000002},"last":4.8659999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.48699999999999999,"15min":0.64900000000000002},"min":{"1min":0.217,"5min":0.214,"15min":0.214},"max":{"1min":4.899,"5min":4.899,"15min":34.948999999999998},"last":4.899}]}]},{"osd":1,"up_from":8,"seq":34359739168,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4645280,"kb_used_data":3444376,"kb_used_omap":676,"kb_used_meta":1200219,"kb_avail":89726560,"statfs":{"total":96636764160,"available":91879997440,"internally_reserved":0,"allocated":3527041024,"data_stored":6791264675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1229024843},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.51300000000000001,"15min":0.44600000000000001},"min":{"1min":0.185,"5min":0.17199999999999999,"15min":0.157},"max":{"1min":3.5070000000000001,"5min":4.0209999999999999,"15min":4.7039999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.42899999999999999,"15min":0.44500000000000001},"min":{"1min":0.20300000000000001,"5min":0.186,"15min":0.186},"max":{"1min":2.4750000000000001,"5min":5.8479999999999999,"15min":7.7510000000000003},"last":0.28100000000000003}]},{"osd":2,"last update":"Fri Oct 20 10:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.40899999999999997,"15min":0.39700000000000002},"min":{"1min":0.216,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.55100000000000005,"5min":3.657,"15min":4.8920000000000003},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.45700000000000002,"15min":0.433},"min":{"1min":0.24099999999999999,"5min":0.23799999999999999,"15min":0.215},"max":{"1min":0.57999999999999996,"5min":3.6869999999999998,"15min":4.9279999999999999},"last":0.48599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739169,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5153140,"kb_used_data":3581548,"kb_used_omap":698,"kb_used_meta":1570885,"kb_avail":89218700,"statfs":{"total":96636764160,"available":91359948800,"internally_reserved":0,"allocated":3667505152,"data_stored":7092708116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":1608586826},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,8,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":1.1930000000000001,"15min":0.72299999999999998},"min":{"1min":0.224,"5min":0.17599999999999999,"15min":0.16700000000000001},"max":{"1min":0.56000000000000005,"5min":81.197000000000003,"15min":81.197000000000003},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.753,"5min":1.2909999999999999,"15min":0.68899999999999995},"min":{"1min":0.22800000000000001,"5min":0.184,"15min":0.184},"max":{"1min":7.2690000000000001,"5min":80.334999999999994,"15min":80.334999999999994},"last":0.22500000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.58099999999999996,"15min":0.52200000000000002},"min":{"1min":0.26900000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.3740000000000001,"5min":11.176,"15min":11.176},"last":0.379},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.64400000000000002,"15min":0.61699999999999999},"min":{"1min":0.23899999999999999,"5min":0.20000000000000001,"15min":0.187},"max":{"1min":3.4209999999999998,"5min":11.093,"15min":11.093},"last":0.29899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3665940480,"data_stored":7092043776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3525435392,"data_stored":6790709267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7191097344,"data_stored":13882474515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:20:03.402 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:20:03.403 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:20:03.403 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:20:03.403 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:20:03.404 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:20:23.405 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:20:23.990 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:20:23.991 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:20:24.032 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2127,"stamp":"2023-10-20T10:20:23.209788+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27152458243,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1804473,"num_read_kb":6976629,"num_write":1763805,"num_write_kb":7053471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69029,"ondisk_log_size":69029,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18597936,"kb_used_data":14105624,"kb_used_omap":2805,"kb_used_meta":4489482,"kb_avail":264517584,"statfs":{"total":289910292480,"available":270866006016,"internally_reserved":0,"allocated":14444158976,"data_stored":27824147174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2873140,"internal_metadata":4597229772},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,7,6,14,10,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14364672,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3486,"num_read_kb":13868,"num_write":3490,"num_write_kb":13960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001797"},"pg_stats":[{"pgid":"2.7","version":"122'9013","reported_seq":13534,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:20:16.398045+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:20:16.398045+0000","last_peered":"2023-10-20T10:20:16.398045+0000","last_clean":"2023-10-20T10:20:16.398045+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:20:16.398045+0000","last_undegraded":"2023-10-20T10:20:16.398045+0000","last_fullsized":"2023-10-20T10:20:16.398045+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9013,"log_dups_size":0,"ondisk_log_size":9013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"122'17705","reported_seq":27743,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:20:16.398733+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:20:16.398733+0000","last_peered":"2023-10-20T10:20:16.398733+0000","last_clean":"2023-10-20T10:20:16.398733+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:20:16.398733+0000","last_undegraded":"2023-10-20T10:20:16.398733+0000","last_fullsized":"2023-10-20T10:20:16.398733+0000","mapping_epoch":13,"log_start":"94'7700","ondisk_log_start":"94'7700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":199,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:55.096661+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:19:55.096661+0000","last_peered":"2023-10-20T10:19:55.096661+0000","last_clean":"2023-10-20T10:19:55.096661+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:19:55.096661+0000","last_undegraded":"2023-10-20T10:19:55.096661+0000","last_fullsized":"2023-10-20T10:19:55.096661+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"122'8976","reported_seq":14271,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:20:16.398077+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:20:16.398077+0000","last_peered":"2023-10-20T10:20:16.398077+0000","last_clean":"2023-10-20T10:20:16.398077+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:20:16.398077+0000","last_undegraded":"2023-10-20T10:20:16.398077+0000","last_fullsized":"2023-10-20T10:20:16.398077+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":8976,"log_dups_size":0,"ondisk_log_size":8976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"122'456387","reported_seq":906276,"reported_epoch":122,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:20:18.373351+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:20:18.373351+0000","last_peered":"2023-10-20T10:20:18.373351+0000","last_clean":"2023-10-20T10:20:18.373351+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:20:18.373351+0000","last_undegraded":"2023-10-20T10:20:18.373351+0000","last_fullsized":"2023-10-20T10:20:18.373351+0000","mapping_epoch":16,"log_start":"121'446330","ondisk_log_start":"121'446330","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":198,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 34s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":6976651264,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":453179,"num_read_kb":1783468,"num_write":452715,"num_write_kb":1810860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":293,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:41.970652+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:19:41.970652+0000","last_peered":"2023-10-20T10:19:41.970652+0000","last_clean":"2023-10-20T10:19:41.970652+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:19:41.970652+0000","last_undegraded":"2023-10-20T10:19:41.970652+0000","last_fullsized":"2023-10-20T10:19:41.970652+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"122'423227","reported_seq":840495,"reported_epoch":122,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:20:19.016069+0000","last_change":"2023-10-20T10:19:59.154102+0000","last_active":"2023-10-20T10:20:19.016069+0000","last_peered":"2023-10-20T10:20:19.016069+0000","last_clean":"2023-10-20T10:20:19.016069+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:20:19.016069+0000","last_undegraded":"2023-10-20T10:20:19.016069+0000","last_fullsized":"2023-10-20T10:20:19.016069+0000","mapping_epoch":16,"log_start":"121'413140","ondisk_log_start":"121'413140","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":155,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 19s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6468362240,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":420334,"num_read_kb":1652088,"num_write":419817,"num_write_kb":1679268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"122'326","reported_seq":6210,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:20:16.402557+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:20:16.402557+0000","last_peered":"2023-10-20T10:20:16.402557+0000","last_clean":"2023-10-20T10:20:16.402557+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:20:16.402557+0000","last_undegraded":"2023-10-20T10:20:16.402557+0000","last_fullsized":"2023-10-20T10:20:16.402557+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"122'449556","reported_seq":893118,"reported_epoch":122,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:20:18.371445+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:20:18.371445+0000","last_peered":"2023-10-20T10:20:18.371445+0000","last_clean":"2023-10-20T10:20:18.371445+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:20:18.371445+0000","last_undegraded":"2023-10-20T10:20:18.371445+0000","last_fullsized":"2023-10-20T10:20:18.371445+0000","mapping_epoch":16,"log_start":"121'439534","ondisk_log_start":"121'439534","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":206,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 33s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":6870355968,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":446869,"num_read_kb":1756904,"num_write":445863,"num_write_kb":1783452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"122'303","reported_seq":6060,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:20:16.398989+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:20:16.398989+0000","last_peered":"2023-10-20T10:20:16.398989+0000","last_clean":"2023-10-20T10:20:16.398989+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:20:16.398989+0000","last_undegraded":"2023-10-20T10:20:16.398989+0000","last_fullsized":"2023-10-20T10:20:16.398989+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"122'447870","reported_seq":890033,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:20:18.999856+0000","last_change":"2023-10-20T10:20:12.098527+0000","last_active":"2023-10-20T10:20:18.999856+0000","last_peered":"2023-10-20T10:20:18.999856+0000","last_clean":"2023-10-20T10:20:18.999856+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:20:18.999856+0000","last_undegraded":"2023-10-20T10:20:18.999856+0000","last_fullsized":"2023-10-20T10:20:18.999856+0000","mapping_epoch":16,"log_start":"121'437824","ondisk_log_start":"121'437824","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6836494355,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":445420,"num_read_kb":1751448,"num_write":444396,"num_write_kb":1777578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":198,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:51.179153+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:19:51.179153+0000","last_peered":"2023-10-20T10:19:51.179153+0000","last_clean":"2023-10-20T10:19:51.179153+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:19:51.179153+0000","last_undegraded":"2023-10-20T10:19:51.179153+0000","last_fullsized":"2023-10-20T10:19:51.179153+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":292,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:42.469749+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:19:42.469749+0000","last_peered":"2023-10-20T10:19:42.469749+0000","last_clean":"2023-10-20T10:19:42.469749+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:19:42.469749+0000","last_undegraded":"2023-10-20T10:19:42.469749+0000","last_fullsized":"2023-10-20T10:19:42.469749+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27151863827,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1765802,"num_read_kb":6943908,"num_write":1762791,"num_write_kb":7051158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14440665088,"data_stored":27823419430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40212,"ondisk_log_size":40212,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":28718,"ondisk_log_size":28718,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739172,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8654556,"kb_used_data":7051348,"kb_used_omap":1430,"kb_used_meta":1601769,"kb_avail":85717284,"statfs":{"total":96636764160,"available":87774498816,"internally_reserved":0,"allocated":7220580352,"data_stored":13911141935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1465289,"internal_metadata":1640211511},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:19:58 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.626,"15min":0.45100000000000001},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.14699999999999999},"max":{"1min":3.9870000000000001,"5min":8.2449999999999992,"15min":8.2449999999999992},"last":0.191},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57699999999999996,"15min":0.501},"min":{"1min":0.215,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.9749999999999996,"5min":6.8239999999999998,"15min":6.8239999999999998},"last":0.27800000000000002}]},{"osd":1,"last update":"Fri Oct 20 10:19:58 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.746,"15min":0.71899999999999997},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":4.8659999999999997,"5min":8.282,"15min":35.670000000000002},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.48699999999999999,"15min":0.64900000000000002},"min":{"1min":0.217,"5min":0.214,"15min":0.214},"max":{"1min":4.899,"5min":4.899,"15min":34.948999999999998},"last":0.315}]}]},{"osd":1,"up_from":8,"seq":34359739172,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4715148,"kb_used_data":3458244,"kb_used_omap":676,"kb_used_meta":1256219,"kb_avail":89656692,"statfs":{"total":96636764160,"available":91808452608,"internally_reserved":0,"allocated":3541241856,"data_stored":6805465507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1286368843},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,6,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.51300000000000001,"15min":0.44600000000000001},"min":{"1min":0.185,"5min":0.17199999999999999,"15min":0.157},"max":{"1min":3.5070000000000001,"5min":4.0209999999999999,"15min":4.7039999999999997},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.42899999999999999,"15min":0.44500000000000001},"min":{"1min":0.20300000000000001,"5min":0.186,"15min":0.186},"max":{"1min":2.4750000000000001,"5min":5.8479999999999999,"15min":7.7510000000000003},"last":0.32200000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.40899999999999997,"15min":0.39700000000000002},"min":{"1min":0.216,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.55100000000000005,"5min":3.657,"15min":4.8920000000000003},"last":0.377},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.45700000000000002,"15min":0.433},"min":{"1min":0.24099999999999999,"5min":0.23799999999999999,"15min":0.215},"max":{"1min":0.57999999999999996,"5min":3.6869999999999998,"15min":4.9279999999999999},"last":0.45300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739173,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5228232,"kb_used_data":3596032,"kb_used_omap":698,"kb_used_meta":1631493,"kb_avail":89143608,"statfs":{"total":96636764160,"available":91283054592,"internally_reserved":0,"allocated":3682336768,"data_stored":7107539732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":1670649418},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,4,2,8,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":1.1930000000000001,"15min":0.72299999999999998},"min":{"1min":0.224,"5min":0.17599999999999999,"15min":0.16700000000000001},"max":{"1min":0.56000000000000005,"5min":81.197000000000003,"15min":81.197000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.753,"5min":1.2909999999999999,"15min":0.68899999999999995},"min":{"1min":0.22800000000000001,"5min":0.184,"15min":0.184},"max":{"1min":7.2690000000000001,"5min":80.334999999999994,"15min":80.334999999999994},"last":0.48099999999999998}]},{"osd":2,"last update":"Fri Oct 20 10:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.58099999999999996,"15min":0.52200000000000002},"min":{"1min":0.26900000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.3740000000000001,"5min":11.176,"15min":11.176},"last":0.501},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.64400000000000002,"15min":0.61699999999999999},"min":{"1min":0.23899999999999999,"5min":0.20000000000000001,"15min":0.187},"max":{"1min":3.4209999999999998,"5min":11.093,"15min":11.093},"last":0.24299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3680575488,"data_stored":7106678784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3539820544,"data_stored":6805094419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7220269056,"data_stored":13911646227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:20:24.034 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:20:24.035 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:20:24.035 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:20:24.036 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:20:24.036 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:20:26.691 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-20T10:20:26.690+0000 7f28abd3f640 -1 calc_pg_upmaps abort due to max <= 0 2023-10-20T10:20:44.037 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:20:44.624 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:20:44.625 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:20:44.652 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2137,"stamp":"2023-10-20T10:20:43.213155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27182117379,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1811743,"num_read_kb":7005553,"num_write":1771045,"num_write_kb":7082431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69093,"ondisk_log_size":69093,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18814420,"kb_used_data":14163900,"kb_used_omap":2805,"kb_used_meta":4647690,"kb_avail":264301100,"statfs":{"total":289910292480,"available":270644326400,"internally_reserved":0,"allocated":14503833600,"data_stored":27883799145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2873140,"internal_metadata":4759234764},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,7,3,8,10,14,15,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15155200,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3692,"num_read_kb":14684,"num_write":3696,"num_write_kb":14784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002056"},"pg_stats":[{"pgid":"2.7","version":"123'9043","reported_seq":13577,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:20:37.706939+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:20:37.706939+0000","last_peered":"2023-10-20T10:20:37.706939+0000","last_clean":"2023-10-20T10:20:37.706939+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:20:37.706939+0000","last_undegraded":"2023-10-20T10:20:37.706939+0000","last_fullsized":"2023-10-20T10:20:37.706939+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9043,"log_dups_size":0,"ondisk_log_size":9043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"123'17765","reported_seq":27828,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:20:37.730775+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:20:37.730775+0000","last_peered":"2023-10-20T10:20:37.730775+0000","last_clean":"2023-10-20T10:20:37.730775+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:20:37.730775+0000","last_undegraded":"2023-10-20T10:20:37.730775+0000","last_fullsized":"2023-10-20T10:20:37.730775+0000","mapping_epoch":13,"log_start":"94'7700","ondisk_log_start":"94'7700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":200,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:55.096749+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:19:55.096749+0000","last_peered":"2023-10-20T10:19:55.096749+0000","last_clean":"2023-10-20T10:19:55.096749+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:19:55.096749+0000","last_undegraded":"2023-10-20T10:19:55.096749+0000","last_fullsized":"2023-10-20T10:19:55.096749+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"123'9006","reported_seq":14314,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:20:37.717340+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:20:37.717340+0000","last_peered":"2023-10-20T10:20:37.717340+0000","last_clean":"2023-10-20T10:20:37.717340+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:20:37.717340+0000","last_undegraded":"2023-10-20T10:20:37.717340+0000","last_fullsized":"2023-10-20T10:20:37.717340+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9006,"log_dups_size":0,"ondisk_log_size":9006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"123'458353","reported_seq":910160,"reported_epoch":123,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:20:38.470595+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:20:38.470595+0000","last_peered":"2023-10-20T10:20:38.470595+0000","last_clean":"2023-10-20T10:20:38.470595+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:20:38.470595+0000","last_undegraded":"2023-10-20T10:20:38.470595+0000","last_fullsized":"2023-10-20T10:20:38.470595+0000","mapping_epoch":16,"log_start":"121'448330","ondisk_log_start":"121'448330","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":315,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 54s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":6984318976,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":455082,"num_read_kb":1791072,"num_write":454591,"num_write_kb":1818364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":294,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:41.970776+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:19:41.970776+0000","last_peered":"2023-10-20T10:19:41.970776+0000","last_clean":"2023-10-20T10:19:41.970776+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:19:41.970776+0000","last_undegraded":"2023-10-20T10:19:41.970776+0000","last_fullsized":"2023-10-20T10:19:41.970776+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"123'425025","reported_seq":844022,"reported_epoch":123,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:20:39.117748+0000","last_change":"2023-10-20T10:19:59.154102+0000","last_active":"2023-10-20T10:20:39.117748+0000","last_peered":"2023-10-20T10:20:39.117748+0000","last_clean":"2023-10-20T10:20:39.117748+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:20:39.117748+0000","last_undegraded":"2023-10-20T10:20:39.117748+0000","last_fullsized":"2023-10-20T10:20:39.117748+0000","mapping_epoch":16,"log_start":"121'414940","ondisk_log_start":"121'414940","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":356,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 39s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6475059200,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":422046,"num_read_kb":1658888,"num_write":421466,"num_write_kb":1685864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"122'326","reported_seq":6228,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:20:36.888850+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:20:36.888850+0000","last_peered":"2023-10-20T10:20:36.888850+0000","last_clean":"2023-10-20T10:20:36.888850+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:20:36.888850+0000","last_undegraded":"2023-10-20T10:20:36.888850+0000","last_fullsized":"2023-10-20T10:20:36.888850+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"123'451479","reported_seq":896869,"reported_epoch":123,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:20:38.463465+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:20:38.463465+0000","last_peered":"2023-10-20T10:20:38.463465+0000","last_clean":"2023-10-20T10:20:38.463465+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:20:38.463465+0000","last_undegraded":"2023-10-20T10:20:38.463465+0000","last_fullsized":"2023-10-20T10:20:38.463465+0000","mapping_epoch":16,"log_start":"121'441434","ondisk_log_start":"121'441434","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":290,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 53s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":6877958144,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":448682,"num_read_kb":1764112,"num_write":447721,"num_write_kb":1790884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"122'303","reported_seq":6077,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:20:36.885540+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:20:36.885540+0000","last_peered":"2023-10-20T10:20:36.885540+0000","last_clean":"2023-10-20T10:20:36.885540+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:20:36.885540+0000","last_undegraded":"2023-10-20T10:20:36.885540+0000","last_fullsized":"2023-10-20T10:20:36.885540+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"123'449727","reported_seq":893736,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:20:39.134360+0000","last_change":"2023-10-20T10:20:35.042588+0000","last_active":"2023-10-20T10:20:39.134360+0000","last_peered":"2023-10-20T10:20:39.134360+0000","last_clean":"2023-10-20T10:20:39.134360+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:20:39.134360+0000","last_undegraded":"2023-10-20T10:20:39.134360+0000","last_fullsized":"2023-10-20T10:20:39.134360+0000","mapping_epoch":16,"log_start":"121'439724","ondisk_log_start":"121'439724","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6844186643,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":447262,"num_read_kb":1758760,"num_write":446253,"num_write_kb":1785006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":199,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:51.179435+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:19:51.179435+0000","last_peered":"2023-10-20T10:19:51.179435+0000","last_clean":"2023-10-20T10:19:51.179435+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:19:51.179435+0000","last_undegraded":"2023-10-20T10:19:51.179435+0000","last_fullsized":"2023-10-20T10:19:51.179435+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":293,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:58.843795+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:19:58.843795+0000","last_peered":"2023-10-20T10:19:58.843795+0000","last_clean":"2023-10-20T10:19:58.843795+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:19:58.843795+0000","last_undegraded":"2023-10-20T10:19:58.843795+0000","last_fullsized":"2023-10-20T10:19:58.843795+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27181522963,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1773072,"num_read_kb":6972832,"num_write":1770031,"num_write_kb":7080118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14500024320,"data_stored":27882778662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40156,"ondisk_log_size":40156,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":28838,"ondisk_log_size":28838,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739176,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8769460,"kb_used_data":7080172,"kb_used_omap":1430,"kb_used_meta":1687849,"kb_avail":85602380,"statfs":{"total":96636764160,"available":87656837120,"internally_reserved":0,"allocated":7250096128,"data_stored":13940650160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1465289,"internal_metadata":1728357431},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:19:58 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.626,"15min":0.45100000000000001},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.14699999999999999},"max":{"1min":3.9870000000000001,"5min":8.2449999999999992,"15min":8.2449999999999992},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57699999999999996,"15min":0.501},"min":{"1min":0.215,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.9749999999999996,"5min":6.8239999999999998,"15min":6.8239999999999998},"last":0.41499999999999998}]},{"osd":1,"last update":"Fri Oct 20 10:19:58 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.746,"15min":0.71899999999999997},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.187},"max":{"1min":4.8659999999999997,"5min":8.282,"15min":35.670000000000002},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.48699999999999999,"15min":0.64900000000000002},"min":{"1min":0.217,"5min":0.214,"15min":0.214},"max":{"1min":4.899,"5min":4.899,"15min":34.948999999999998},"last":0.34200000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739176,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4765900,"kb_used_data":3472900,"kb_used_omap":676,"kb_used_meta":1292315,"kb_avail":89605940,"statfs":{"total":96636764160,"available":91756482560,"internally_reserved":0,"allocated":3556249600,"data_stored":6820465700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1323331147},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,6,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.51300000000000001,"15min":0.44600000000000001},"min":{"1min":0.185,"5min":0.17199999999999999,"15min":0.157},"max":{"1min":3.5070000000000001,"5min":4.0209999999999999,"15min":4.7039999999999997},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.42899999999999999,"15min":0.44500000000000001},"min":{"1min":0.20300000000000001,"5min":0.186,"15min":0.186},"max":{"1min":2.4750000000000001,"5min":5.8479999999999999,"15min":7.7510000000000003},"last":0.22900000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:19:41 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.40899999999999997,"15min":0.39700000000000002},"min":{"1min":0.216,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.55100000000000005,"5min":3.657,"15min":4.8920000000000003},"last":0.311},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.45700000000000002,"15min":0.433},"min":{"1min":0.24099999999999999,"5min":0.23799999999999999,"15min":0.215},"max":{"1min":0.57999999999999996,"5min":3.6869999999999998,"15min":4.9279999999999999},"last":0.34300000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739177,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5279060,"kb_used_data":3610828,"kb_used_omap":698,"kb_used_meta":1667525,"kb_avail":89092780,"statfs":{"total":96636764160,"available":91231006720,"internally_reserved":0,"allocated":3697487872,"data_stored":7122683285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":1707546186},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,5,11,13,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":1.1930000000000001,"15min":0.72299999999999998},"min":{"1min":0.224,"5min":0.17599999999999999,"15min":0.16700000000000001},"max":{"1min":0.56000000000000005,"5min":81.197000000000003,"15min":81.197000000000003},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.753,"5min":1.2909999999999999,"15min":0.68899999999999995},"min":{"1min":0.22800000000000001,"5min":0.184,"15min":0.184},"max":{"1min":7.2690000000000001,"5min":80.334999999999994,"15min":80.334999999999994},"last":0.24199999999999999}]},{"osd":2,"last update":"Fri Oct 20 10:19:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.58099999999999996,"15min":0.52200000000000002},"min":{"1min":0.26900000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.3740000000000001,"5min":11.176,"15min":11.176},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.64400000000000002,"15min":0.61699999999999999},"min":{"1min":0.23899999999999999,"5min":0.20000000000000001,"15min":0.187},"max":{"1min":3.4209999999999998,"5min":11.093,"15min":11.093},"last":0.34100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3695882240,"data_stored":7121985536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3554177024,"data_stored":6819450899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7249965056,"data_stored":13941342227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:20:44.653 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:20:44.654 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:20:44.654 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:20:44.655 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:20:44.655 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:21:04.656 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:21:05.234 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:21:05.235 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:21:05.269 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2148,"stamp":"2023-10-20T10:21:05.216935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27217535491,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1820401,"num_read_kb":7040037,"num_write":1779724,"num_write_kb":7117147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69248,"ondisk_log_size":69248,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19083564,"kb_used_data":14233748,"kb_used_omap":2805,"kb_used_meta":4846986,"kb_avail":264031956,"statfs":{"total":289910292480,"available":270368722944,"internally_reserved":0,"allocated":14575357952,"data_stored":27955323497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2873140,"internal_metadata":4963313868},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,6,7,18,17],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21250048,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5173,"num_read_kb":20608,"num_write":5207,"num_write_kb":20828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002048"},"pg_stats":[{"pgid":"2.7","version":"123'9103","reported_seq":13652,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:21:02.482081+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:21:02.482081+0000","last_peered":"2023-10-20T10:21:02.482081+0000","last_clean":"2023-10-20T10:21:02.482081+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:21:02.482081+0000","last_undegraded":"2023-10-20T10:21:02.482081+0000","last_fullsized":"2023-10-20T10:21:02.482081+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9103,"log_dups_size":0,"ondisk_log_size":9103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"123'17885","reported_seq":27979,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:21:02.483262+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:21:02.483262+0000","last_peered":"2023-10-20T10:21:02.483262+0000","last_clean":"2023-10-20T10:21:02.483262+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:21:02.483262+0000","last_undegraded":"2023-10-20T10:21:02.483262+0000","last_fullsized":"2023-10-20T10:21:02.483262+0000","mapping_epoch":13,"log_start":"94'7800","ondisk_log_start":"94'7800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":201,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:20:23.013374+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:20:23.013374+0000","last_peered":"2023-10-20T10:20:23.013374+0000","last_clean":"2023-10-20T10:20:23.013374+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:20:23.013374+0000","last_undegraded":"2023-10-20T10:20:23.013374+0000","last_fullsized":"2023-10-20T10:20:23.013374+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"123'9066","reported_seq":14390,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:21:02.482171+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:21:02.482171+0000","last_peered":"2023-10-20T10:21:02.482171+0000","last_clean":"2023-10-20T10:21:02.482171+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:21:02.482171+0000","last_undegraded":"2023-10-20T10:21:02.482171+0000","last_fullsized":"2023-10-20T10:21:02.482171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9066,"log_dups_size":0,"ondisk_log_size":9066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"123'460788","reported_seq":914790,"reported_epoch":123,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:21:03.603020+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:21:03.603020+0000","last_peered":"2023-10-20T10:21:03.603020+0000","last_clean":"2023-10-20T10:21:03.603020+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:21:03.603020+0000","last_undegraded":"2023-10-20T10:21:03.603020+0000","last_fullsized":"2023-10-20T10:21:03.603020+0000","mapping_epoch":16,"log_start":"122'450730","ondisk_log_start":"122'450730","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":549,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 79s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":6993530880,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":457265,"num_read_kb":1799764,"num_write":456849,"num_write_kb":1827396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":295,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:20:23.020527+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:20:23.020527+0000","last_peered":"2023-10-20T10:20:23.020527+0000","last_clean":"2023-10-20T10:20:23.020527+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:20:23.020527+0000","last_undegraded":"2023-10-20T10:20:23.020527+0000","last_fullsized":"2023-10-20T10:20:23.020527+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"123'427293","reported_seq":848378,"reported_epoch":123,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:21:04.204518+0000","last_change":"2023-10-20T10:19:59.154102+0000","last_active":"2023-10-20T10:21:04.204518+0000","last_peered":"2023-10-20T10:21:04.204518+0000","last_clean":"2023-10-20T10:21:04.204518+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:21:04.204518+0000","last_undegraded":"2023-10-20T10:21:04.204518+0000","last_fullsized":"2023-10-20T10:21:04.204518+0000","mapping_epoch":16,"log_start":"122'417240","ondisk_log_start":"122'417240","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":593,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 65s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6483591168,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":424118,"num_read_kb":1667128,"num_write":423548,"num_write_kb":1694192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"122'326","reported_seq":6248,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:21:02.557936+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:21:02.557936+0000","last_peered":"2023-10-20T10:21:02.557936+0000","last_clean":"2023-10-20T10:21:02.557936+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:21:02.557936+0000","last_undegraded":"2023-10-20T10:21:02.557936+0000","last_fullsized":"2023-10-20T10:21:02.557936+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"123'453848","reported_seq":901434,"reported_epoch":123,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:21:03.588369+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:21:03.588369+0000","last_peered":"2023-10-20T10:21:03.588369+0000","last_clean":"2023-10-20T10:21:03.588369+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:21:03.588369+0000","last_undegraded":"2023-10-20T10:21:03.588369+0000","last_fullsized":"2023-10-20T10:21:03.588369+0000","mapping_epoch":16,"log_start":"122'443834","ondisk_log_start":"122'443834","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":513,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 78s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":6886903808,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":450864,"num_read_kb":1772832,"num_write":449917,"num_write_kb":1799668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"122'303","reported_seq":6097,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:21:02.496388+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:21:02.496388+0000","last_peered":"2023-10-20T10:21:02.496388+0000","last_clean":"2023-10-20T10:21:02.496388+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:21:02.496388+0000","last_undegraded":"2023-10-20T10:21:02.496388+0000","last_fullsized":"2023-10-20T10:21:02.496388+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"123'451870","reported_seq":898099,"reported_epoch":123,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:21:04.232130+0000","last_change":"2023-10-20T10:21:04.232244+0000","last_active":"2023-10-20T10:21:04.232130+0000","last_peered":"2023-10-20T10:21:04.232130+0000","last_clean":"2023-10-20T10:21:04.232130+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:21:04.232130+0000","last_undegraded":"2023-10-20T10:21:04.232130+0000","last_fullsized":"2023-10-20T10:21:04.232130+0000","mapping_epoch":16,"log_start":"122'441824","ondisk_log_start":"122'441824","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6852915219,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":449483,"num_read_kb":1767592,"num_write":448396,"num_write_kb":1793578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":200,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:20:23.013398+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:20:23.013398+0000","last_peered":"2023-10-20T10:20:23.013398+0000","last_clean":"2023-10-20T10:20:23.013398+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:20:23.013398+0000","last_undegraded":"2023-10-20T10:20:23.013398+0000","last_fullsized":"2023-10-20T10:20:23.013398+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":293,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:58.843795+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:19:58.843795+0000","last_peered":"2023-10-20T10:19:58.843795+0000","last_clean":"2023-10-20T10:19:58.843795+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:19:58.843795+0000","last_undegraded":"2023-10-20T10:19:58.843795+0000","last_fullsized":"2023-10-20T10:19:58.843795+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27216941075,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1781730,"num_read_kb":7007316,"num_write":1778710,"num_write_kb":7114834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14571143168,"data_stored":27953897510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40171,"ondisk_log_size":40171,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":28978,"ondisk_log_size":28978,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739181,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8881688,"kb_used_data":7116048,"kb_used_omap":1430,"kb_used_meta":1764201,"kb_avail":85490152,"statfs":{"total":96636764160,"available":87541915648,"internally_reserved":0,"allocated":7286833152,"data_stored":13977387184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1465289,"internal_metadata":1806541879},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:21:00 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.624,"15min":0.44900000000000001},"min":{"1min":0.16700000000000001,"5min":0.16200000000000001,"15min":0.14699999999999999},"max":{"1min":0.58299999999999996,"5min":8.2449999999999992,"15min":8.2449999999999992},"last":0.16800000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.57799999999999996,"15min":0.499},"min":{"1min":0.23000000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.51300000000000001,"5min":6.8239999999999998,"15min":6.8239999999999998},"last":0.27200000000000002}]},{"osd":1,"last update":"Fri Oct 20 10:21:00 2023","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.73599999999999999,"15min":0.71799999999999997},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":0.77400000000000002,"5min":8.282,"15min":35.670000000000002},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.504,"15min":0.65100000000000002},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.214},"max":{"1min":1.486,"5min":4.899,"15min":34.948999999999998},"last":0.30199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739181,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4840416,"kb_used_data":3489112,"kb_used_omap":676,"kb_used_meta":1350619,"kb_avail":89531424,"statfs":{"total":96636764160,"available":91680178176,"internally_reserved":0,"allocated":3572850688,"data_stored":6837066788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1383034443},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.501,"15min":0.44400000000000001},"min":{"1min":0.20300000000000001,"5min":0.17199999999999999,"15min":0.157},"max":{"1min":0.69499999999999995,"5min":4.0209999999999999,"15min":4.7039999999999997},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.26300000000000001,"5min":0.42599999999999999,"15min":0.439},"min":{"1min":0.21099999999999999,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":0.34999999999999998,"5min":5.8479999999999999,"15min":7.7510000000000003},"last":0.222}]},{"osd":2,"last update":"Fri Oct 20 10:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.40600000000000003,"15min":0.39400000000000002},"min":{"1min":0.19900000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.44400000000000001,"5min":3.657,"15min":4.8920000000000003},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.44900000000000001,"15min":0.435},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.215},"max":{"1min":0.58499999999999996,"5min":3.6869999999999998,"15min":4.9279999999999999},"last":0.35299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739182,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5361460,"kb_used_data":3628588,"kb_used_omap":698,"kb_used_meta":1732165,"kb_avail":89010380,"statfs":{"total":96636764160,"available":91146629120,"internally_reserved":0,"allocated":3715674112,"data_stored":7140869525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":1773737546},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,5,13,16],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.46999999999999997,"15min":0.73499999999999999},"min":{"1min":0.20300000000000001,"5min":0.17599999999999999,"15min":0.16700000000000001},"max":{"1min":3.6920000000000002,"5min":9.266,"15min":81.197000000000003},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51200000000000001,"15min":0.68100000000000005},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.184},"max":{"1min":3.7170000000000001,"5min":7.2690000000000001,"15min":80.334999999999994},"last":0.27400000000000002}]},{"osd":2,"last update":"Fri Oct 20 10:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.61699999999999999,"15min":0.53300000000000003},"min":{"1min":0.251,"5min":0.23899999999999999,"15min":0.20899999999999999},"max":{"1min":3.653,"5min":11.176,"15min":11.176},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.63700000000000001,"15min":0.61099999999999999},"min":{"1min":0.22600000000000001,"5min":0.20000000000000001,"15min":0.187},"max":{"1min":0.63200000000000001,"5min":11.093,"15min":11.093},"last":0.245}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3714113536,"data_stored":7140216832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3571486720,"data_stored":6836760595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7285542912,"data_stored":13976920083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:21:05.271 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:21:05.271 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:21:05.271 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:21:05.272 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:21:05.272 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:21:25.274 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:21:25.722 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:21:25.723 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:21:25.771 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2158,"stamp":"2023-10-20T10:21:25.220364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27247550979,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1827768,"num_read_kb":7069373,"num_write":1787068,"num_write_kb":7146523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69268,"ondisk_log_size":69268,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19278372,"kb_used_data":14292492,"kb_used_omap":2805,"kb_used_meta":4983050,"kb_avail":263837148,"statfs":{"total":289910292480,"available":270169239552,"internally_reserved":0,"allocated":14635511808,"data_stored":28015477353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2873140,"internal_metadata":5102643404},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,8,14,11,18,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22327296,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5494,"num_read_kb":21884,"num_write":5464,"num_write_kb":21856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002030"},"pg_stats":[{"pgid":"2.7","version":"123'9118","reported_seq":13680,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:21:22.962036+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:21:22.962036+0000","last_peered":"2023-10-20T10:21:22.962036+0000","last_clean":"2023-10-20T10:21:22.962036+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:21:22.962036+0000","last_undegraded":"2023-10-20T10:21:22.962036+0000","last_fullsized":"2023-10-20T10:21:22.962036+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9118,"log_dups_size":0,"ondisk_log_size":9118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"123'17915","reported_seq":28033,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:21:22.962640+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:21:22.962640+0000","last_peered":"2023-10-20T10:21:22.962640+0000","last_clean":"2023-10-20T10:21:22.962640+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:21:22.962640+0000","last_undegraded":"2023-10-20T10:21:22.962640+0000","last_fullsized":"2023-10-20T10:21:22.962640+0000","mapping_epoch":13,"log_start":"94'7900","ondisk_log_start":"94'7900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":201,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:20:23.013374+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:20:23.013374+0000","last_peered":"2023-10-20T10:20:23.013374+0000","last_clean":"2023-10-20T10:20:23.013374+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:20:23.013374+0000","last_undegraded":"2023-10-20T10:20:23.013374+0000","last_fullsized":"2023-10-20T10:20:23.013374+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"123'9081","reported_seq":14417,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:21:22.962008+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:21:22.962008+0000","last_peered":"2023-10-20T10:21:22.962008+0000","last_clean":"2023-10-20T10:21:22.962008+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:21:22.962008+0000","last_undegraded":"2023-10-20T10:21:22.962008+0000","last_fullsized":"2023-10-20T10:21:22.962008+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9081,"log_dups_size":0,"ondisk_log_size":9081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"123'462790","reported_seq":918634,"reported_epoch":123,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:21:23.704847+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:21:23.704847+0000","last_peered":"2023-10-20T10:21:23.704847+0000","last_clean":"2023-10-20T10:21:23.704847+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:21:23.704847+0000","last_undegraded":"2023-10-20T10:21:23.704847+0000","last_fullsized":"2023-10-20T10:21:23.704847+0000","mapping_epoch":16,"log_start":"122'452730","ondisk_log_start":"122'452730","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":683,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 99s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7001280512,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":459094,"num_read_kb":1807036,"num_write":458751,"num_write_kb":1835004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":295,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:20:23.020527+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:20:23.020527+0000","last_peered":"2023-10-20T10:20:23.020527+0000","last_clean":"2023-10-20T10:20:23.020527+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:20:23.020527+0000","last_undegraded":"2023-10-20T10:20:23.020527+0000","last_fullsized":"2023-10-20T10:20:23.020527+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"123'429134","reported_seq":852027,"reported_epoch":123,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:21:24.293611+0000","last_change":"2023-10-20T10:19:59.154102+0000","last_active":"2023-10-20T10:21:24.293611+0000","last_peered":"2023-10-20T10:21:24.293611+0000","last_clean":"2023-10-20T10:21:24.293611+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:21:24.293611+0000","last_undegraded":"2023-10-20T10:21:24.293611+0000","last_fullsized":"2023-10-20T10:21:24.293611+0000","mapping_epoch":16,"log_start":"122'419040","ondisk_log_start":"122'419040","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":721,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 85s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6490738688,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":425911,"num_read_kb":1674280,"num_write":425293,"num_write_kb":1701172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"122'326","reported_seq":6264,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:21:23.007682+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:21:23.007682+0000","last_peered":"2023-10-20T10:21:23.007682+0000","last_clean":"2023-10-20T10:21:23.007682+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:21:23.007682+0000","last_undegraded":"2023-10-20T10:21:23.007682+0000","last_fullsized":"2023-10-20T10:21:23.007682+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"123'455877","reported_seq":905352,"reported_epoch":123,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:21:23.696699+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:21:23.696699+0000","last_peered":"2023-10-20T10:21:23.696699+0000","last_clean":"2023-10-20T10:21:23.696699+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:21:23.696699+0000","last_undegraded":"2023-10-20T10:21:23.696699+0000","last_fullsized":"2023-10-20T10:21:23.696699+0000","mapping_epoch":16,"log_start":"122'445834","ondisk_log_start":"122'445834","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":669,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 98s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":6894723072,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":452742,"num_read_kb":1780312,"num_write":451826,"num_write_kb":1807304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"122'303","reported_seq":6113,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:21:22.975054+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:21:22.975054+0000","last_peered":"2023-10-20T10:21:22.975054+0000","last_clean":"2023-10-20T10:21:22.975054+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:21:22.975054+0000","last_undegraded":"2023-10-20T10:21:22.975054+0000","last_fullsized":"2023-10-20T10:21:22.975054+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"123'453658","reported_seq":901767,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:21:24.329578+0000","last_change":"2023-10-20T10:21:24.116924+0000","last_active":"2023-10-20T10:21:24.329578+0000","last_peered":"2023-10-20T10:21:24.329578+0000","last_clean":"2023-10-20T10:21:24.329578+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:21:24.329578+0000","last_undegraded":"2023-10-20T10:21:24.329578+0000","last_fullsized":"2023-10-20T10:21:24.329578+0000","mapping_epoch":16,"log_start":"122'443624","ondisk_log_start":"122'443624","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6860214291,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":451350,"num_read_kb":1775024,"num_write":450184,"num_write_kb":1800730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":200,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:20:23.013398+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:20:23.013398+0000","last_peered":"2023-10-20T10:20:23.013398+0000","last_clean":"2023-10-20T10:20:23.013398+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:20:23.013398+0000","last_undegraded":"2023-10-20T10:20:23.013398+0000","last_fullsized":"2023-10-20T10:20:23.013398+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":293,"reported_epoch":122,"state":"active+clean","last_fresh":"2023-10-20T10:19:58.843795+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:19:58.843795+0000","last_peered":"2023-10-20T10:19:58.843795+0000","last_clean":"2023-10-20T10:19:58.843795+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:19:58.843795+0000","last_undegraded":"2023-10-20T10:19:58.843795+0000","last_fullsized":"2023-10-20T10:19:58.843795+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27246956563,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1789097,"num_read_kb":7036652,"num_write":1786054,"num_write_kb":7144210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14631284736,"data_stored":28014039078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40231,"ondisk_log_size":40231,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":28938,"ondisk_log_size":28938,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739185,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8978604,"kb_used_data":7144932,"kb_used_omap":1430,"kb_used_meta":1832233,"kb_avail":85393236,"statfs":{"total":96636764160,"available":87442673664,"internally_reserved":0,"allocated":7316410368,"data_stored":14006964400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1465289,"internal_metadata":1876206647},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:21:00 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.624,"15min":0.44900000000000001},"min":{"1min":0.16700000000000001,"5min":0.16200000000000001,"15min":0.14699999999999999},"max":{"1min":0.58299999999999996,"5min":8.2449999999999992,"15min":8.2449999999999992},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.57799999999999996,"15min":0.499},"min":{"1min":0.23000000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.51300000000000001,"5min":6.8239999999999998,"15min":6.8239999999999998},"last":0.20499999999999999}]},{"osd":1,"last update":"Fri Oct 20 10:21:00 2023","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.73599999999999999,"15min":0.71799999999999997},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":0.77400000000000002,"5min":8.282,"15min":35.670000000000002},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.432,"5min":0.504,"15min":0.65100000000000002},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.214},"max":{"1min":1.486,"5min":4.899,"15min":34.948999999999998},"last":0.23799999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739185,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4887052,"kb_used_data":3503812,"kb_used_omap":676,"kb_used_meta":1382555,"kb_avail":89484788,"statfs":{"total":96636764160,"available":91632422912,"internally_reserved":0,"allocated":3587903488,"data_stored":6852119588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1415736907},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,5,9,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.501,"15min":0.44400000000000001},"min":{"1min":0.20300000000000001,"5min":0.17199999999999999,"15min":0.157},"max":{"1min":0.69499999999999995,"5min":4.0209999999999999,"15min":4.7039999999999997},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.26300000000000001,"5min":0.42599999999999999,"15min":0.439},"min":{"1min":0.21099999999999999,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":0.34999999999999998,"5min":5.8479999999999999,"15min":7.7510000000000003},"last":0.23400000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.40600000000000003,"15min":0.39400000000000002},"min":{"1min":0.19900000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.44400000000000001,"5min":3.657,"15min":4.8920000000000003},"last":0.314},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.44900000000000001,"15min":0.435},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.215},"max":{"1min":0.58499999999999996,"5min":3.6869999999999998,"15min":4.9279999999999999},"last":0.33600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739186,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5412716,"kb_used_data":3643748,"kb_used_omap":698,"kb_used_meta":1768261,"kb_avail":88959124,"statfs":{"total":96636764160,"available":91094142976,"internally_reserved":0,"allocated":3731197952,"data_stored":7156393365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":1810699850},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,8,3,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.46999999999999997,"15min":0.73499999999999999},"min":{"1min":0.20300000000000001,"5min":0.17599999999999999,"15min":0.16700000000000001},"max":{"1min":3.6920000000000002,"5min":9.266,"15min":81.197000000000003},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51200000000000001,"15min":0.68100000000000005},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.184},"max":{"1min":3.7170000000000001,"5min":7.2690000000000001,"15min":80.334999999999994},"last":0.501}]},{"osd":2,"last update":"Fri Oct 20 10:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.61699999999999999,"15min":0.53300000000000003},"min":{"1min":0.251,"5min":0.23899999999999999,"15min":0.20899999999999999},"max":{"1min":3.653,"5min":11.176,"15min":11.176},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.63700000000000001,"15min":0.61099999999999999},"min":{"1min":0.22600000000000001,"5min":0.20000000000000001,"15min":0.187},"max":{"1min":0.63200000000000001,"5min":11.093,"15min":11.093},"last":0.307}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3729731584,"data_stored":7155834880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3585966080,"data_stored":6851239955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7315587072,"data_stored":14006964243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:21:25.773 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:21:25.774 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:21:25.774 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:21:25.775 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:21:25.775 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:21:45.776 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:21:46.265 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:21:46.266 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:21:46.313 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2169,"stamp":"2023-10-20T10:21:45.225189+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27275936259,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1834713,"num_read_kb":7097061,"num_write":1793994,"num_write_kb":7174227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69331,"ondisk_log_size":69331,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19544456,"kb_used_data":14348080,"kb_used_omap":2805,"kb_used_meta":5193546,"kb_avail":263571064,"statfs":{"total":289910292480,"available":269896769536,"internally_reserved":0,"allocated":14692433920,"data_stored":28072376683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2873140,"internal_metadata":5318191308},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,5,3,2,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21012480,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5172,"num_read_kb":20620,"num_write":5125,"num_write_kb":20500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003363"},"pg_stats":[{"pgid":"2.7","version":"124'9142","reported_seq":13717,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-20T10:21:43.438130+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:21:43.438130+0000","last_peered":"2023-10-20T10:21:43.438130+0000","last_clean":"2023-10-20T10:21:43.438130+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:21:43.438130+0000","last_undegraded":"2023-10-20T10:21:43.438130+0000","last_fullsized":"2023-10-20T10:21:43.438130+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9142,"log_dups_size":0,"ondisk_log_size":9142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"124'17963","reported_seq":28106,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-20T10:21:43.438735+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:21:43.438735+0000","last_peered":"2023-10-20T10:21:43.438735+0000","last_clean":"2023-10-20T10:21:43.438735+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:21:43.438735+0000","last_undegraded":"2023-10-20T10:21:43.438735+0000","last_fullsized":"2023-10-20T10:21:43.438735+0000","mapping_epoch":13,"log_start":"94'7900","ondisk_log_start":"94'7900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":202,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:20:44.142687+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:20:44.142687+0000","last_peered":"2023-10-20T10:20:44.142687+0000","last_clean":"2023-10-20T10:20:44.142687+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:20:44.142687+0000","last_undegraded":"2023-10-20T10:20:44.142687+0000","last_fullsized":"2023-10-20T10:20:44.142687+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"124'9105","reported_seq":14454,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-20T10:21:43.438683+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:21:43.438683+0000","last_peered":"2023-10-20T10:21:43.438683+0000","last_clean":"2023-10-20T10:21:43.438683+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:21:43.438683+0000","last_undegraded":"2023-10-20T10:21:43.438683+0000","last_fullsized":"2023-10-20T10:21:43.438683+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9105,"log_dups_size":0,"ondisk_log_size":9105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"124'464614","reported_seq":922250,"reported_epoch":124,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:21:43.828648+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:21:43.828648+0000","last_peered":"2023-10-20T10:21:43.828648+0000","last_clean":"2023-10-20T10:21:43.828648+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:21:43.828648+0000","last_undegraded":"2023-10-20T10:21:43.828648+0000","last_fullsized":"2023-10-20T10:21:43.828648+0000","mapping_epoch":16,"log_start":"122'454530","ondisk_log_start":"122'454530","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":814,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 119s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7008313344,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":460872,"num_read_kb":1814132,"num_write":460475,"num_write_kb":1841900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":296,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:20:43.489246+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:20:43.489246+0000","last_peered":"2023-10-20T10:20:43.489246+0000","last_clean":"2023-10-20T10:20:43.489246+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:20:43.489246+0000","last_undegraded":"2023-10-20T10:20:43.489246+0000","last_fullsized":"2023-10-20T10:20:43.489246+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"124'430910","reported_seq":855452,"reported_epoch":124,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:21:44.423392+0000","last_change":"2023-10-20T10:19:59.154102+0000","last_active":"2023-10-20T10:21:44.423392+0000","last_peered":"2023-10-20T10:21:44.423392+0000","last_clean":"2023-10-20T10:21:44.423392+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:21:44.423392+0000","last_undegraded":"2023-10-20T10:21:44.423392+0000","last_fullsized":"2023-10-20T10:21:44.423392+0000","mapping_epoch":16,"log_start":"122'420840","ondisk_log_start":"122'420840","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":858,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 105s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6497595392,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":427548,"num_read_kb":1680800,"num_write":426967,"num_write_kb":1707868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"122'326","reported_seq":6281,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-20T10:21:43.439676+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:21:43.439676+0000","last_peered":"2023-10-20T10:21:43.439676+0000","last_clean":"2023-10-20T10:21:43.439676+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:21:43.439676+0000","last_undegraded":"2023-10-20T10:21:43.439676+0000","last_fullsized":"2023-10-20T10:21:43.439676+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"124'457837","reported_seq":909135,"reported_epoch":124,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:21:43.804678+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:21:43.804678+0000","last_peered":"2023-10-20T10:21:43.804678+0000","last_clean":"2023-10-20T10:21:43.804678+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:21:43.804678+0000","last_undegraded":"2023-10-20T10:21:43.804678+0000","last_fullsized":"2023-10-20T10:21:43.804678+0000","mapping_epoch":16,"log_start":"122'447834","ondisk_log_start":"122'447834","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":853,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 118s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":6902198272,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":454550,"num_read_kb":1787508,"num_write":453647,"num_write_kb":1814588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"122'303","reported_seq":6130,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-20T10:21:43.441556+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:21:43.441556+0000","last_peered":"2023-10-20T10:21:43.441556+0000","last_clean":"2023-10-20T10:21:43.441556+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:21:43.441556+0000","last_undegraded":"2023-10-20T10:21:43.441556+0000","last_fullsized":"2023-10-20T10:21:43.441556+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"124'455365","reported_seq":905204,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-20T10:21:44.423858+0000","last_change":"2023-10-20T10:21:43.261933+0000","last_active":"2023-10-20T10:21:44.423858+0000","last_peered":"2023-10-20T10:21:44.423858+0000","last_clean":"2023-10-20T10:21:44.423858+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:21:44.423858+0000","last_undegraded":"2023-10-20T10:21:44.423858+0000","last_fullsized":"2023-10-20T10:21:44.423858+0000","mapping_epoch":16,"log_start":"122'445324","ondisk_log_start":"122'445324","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6867234835,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":453072,"num_read_kb":1781900,"num_write":451891,"num_write_kb":1807558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":201,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:20:44.142657+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:20:44.142657+0000","last_peered":"2023-10-20T10:20:44.142657+0000","last_clean":"2023-10-20T10:20:44.142657+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:20:44.142657+0000","last_undegraded":"2023-10-20T10:20:44.142657+0000","last_fullsized":"2023-10-20T10:20:44.142657+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":294,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:20:23.018388+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:20:23.018388+0000","last_peered":"2023-10-20T10:20:23.018388+0000","last_clean":"2023-10-20T10:20:23.018388+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:20:23.018388+0000","last_undegraded":"2023-10-20T10:20:23.018388+0000","last_fullsized":"2023-10-20T10:20:23.018388+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27275341843,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1796042,"num_read_kb":7064340,"num_write":1792980,"num_write_kb":7171914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14688092160,"data_stored":28070846502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40198,"ondisk_log_size":40198,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":29034,"ondisk_log_size":29034,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739189,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9120076,"kb_used_data":7172548,"kb_used_omap":1430,"kb_used_meta":1946089,"kb_avail":85251764,"statfs":{"total":96636764160,"available":87297806336,"internally_reserved":0,"allocated":7344689152,"data_stored":14035235590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1465289,"internal_metadata":1992795191},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:21:00 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.624,"15min":0.44900000000000001},"min":{"1min":0.16700000000000001,"5min":0.16200000000000001,"15min":0.14699999999999999},"max":{"1min":0.58299999999999996,"5min":8.2449999999999992,"15min":8.2449999999999992},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.57799999999999996,"15min":0.499},"min":{"1min":0.23000000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.51300000000000001,"5min":6.8239999999999998,"15min":6.8239999999999998},"last":0.32800000000000001}]},{"osd":1,"last update":"Fri Oct 20 10:21:00 2023","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.73599999999999999,"15min":0.71799999999999997},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":0.77400000000000002,"5min":8.282,"15min":35.670000000000002},"last":1.6879999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.504,"15min":0.65100000000000002},"min":{"1min":0.22500000000000001,"5min":0.214,"15min":0.214},"max":{"1min":1.486,"5min":4.899,"15min":34.948999999999998},"last":0.35899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739189,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4936764,"kb_used_data":3517428,"kb_used_omap":676,"kb_used_meta":1418651,"kb_avail":89435076,"statfs":{"total":96636764160,"available":91581517824,"internally_reserved":0,"allocated":3601846272,"data_stored":6866054778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1452699211},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.51200000000000001,"15min":0.441},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.157},"max":{"1min":1.4330000000000001,"5min":4.0209999999999999,"15min":4.7039999999999997},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.40000000000000002,"15min":0.41899999999999998},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":1.5,"5min":5.8479999999999999,"15min":5.8479999999999999},"last":0.25800000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40600000000000003,"15min":0.38500000000000001},"min":{"1min":0.23899999999999999,"5min":0.19900000000000001,"15min":0.18099999999999999},"max":{"1min":0.378,"5min":3.657,"15min":4.8920000000000003},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.44600000000000001,"15min":0.42699999999999999},"min":{"1min":0.27500000000000002,"5min":0.23999999999999999,"15min":0.222},"max":{"1min":0.40400000000000003,"5min":3.6869999999999998,"15min":4.9279999999999999},"last":0.36399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739190,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5487616,"kb_used_data":3658104,"kb_used_omap":698,"kb_used_meta":1828805,"kb_avail":88884224,"statfs":{"total":96636764160,"available":91017445376,"internally_reserved":0,"allocated":3745898496,"data_stored":7171086315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":1872696906},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.46999999999999997,"15min":0.73699999999999999},"min":{"1min":0.19,"5min":0.17599999999999999,"15min":0.16700000000000001},"max":{"1min":0.50900000000000001,"5min":9.266,"15min":81.197000000000003},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.51900000000000002,"15min":0.68600000000000005},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.184},"max":{"1min":2.1400000000000001,"5min":7.2690000000000001,"15min":80.334999999999994},"last":0.39400000000000002}]},{"osd":2,"last update":"Fri Oct 20 10:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.621,"15min":0.53200000000000003},"min":{"1min":0.23999999999999999,"5min":0.23899999999999999,"15min":0.20899999999999999},"max":{"1min":0.56399999999999995,"5min":11.176,"15min":11.176},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.64900000000000002,"15min":0.61399999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.187},"max":{"1min":1.5049999999999999,"5min":11.093,"15min":11.093},"last":0.307}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3744251904,"data_stored":7170355200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3599798272,"data_stored":6865072147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7344041984,"data_stored":14035419155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:21:46.315 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:21:46.316 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:21:46.316 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:21:46.317 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:21:46.317 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:22:06.056 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:22:06.055+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:22:06.057 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:22:06.055+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:22:06.318 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:22:06.862 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:22:06.863 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:22:06.892 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2179,"stamp":"2023-10-20T10:22:05.228623+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27304473091,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1841737,"num_read_kb":7125057,"num_write":1800962,"num_write_kb":7202099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69392,"ondisk_log_size":69392,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19609980,"kb_used_data":14403492,"kb_used_omap":2805,"kb_used_meta":5203658,"kb_avail":263505540,"statfs":{"total":289910292480,"available":269829672960,"internally_reserved":0,"allocated":14749175808,"data_stored":28129118571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2873140,"internal_metadata":5328545996},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,10,19,12,12,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21131264,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5184,"num_read_kb":20664,"num_write":5153,"num_write_kb":20612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002023"},"pg_stats":[{"pgid":"2.7","version":"124'9178","reported_seq":13765,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-20T10:22:04.463198+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:22:04.463198+0000","last_peered":"2023-10-20T10:22:04.463198+0000","last_clean":"2023-10-20T10:22:04.463198+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:22:04.463198+0000","last_undegraded":"2023-10-20T10:22:04.463198+0000","last_fullsized":"2023-10-20T10:22:04.463198+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9178,"log_dups_size":0,"ondisk_log_size":9178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"124'18035","reported_seq":28202,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-20T10:22:04.484318+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:22:04.484318+0000","last_peered":"2023-10-20T10:22:04.484318+0000","last_clean":"2023-10-20T10:22:04.484318+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:22:04.484318+0000","last_undegraded":"2023-10-20T10:22:04.484318+0000","last_fullsized":"2023-10-20T10:22:04.484318+0000","mapping_epoch":13,"log_start":"94'8000","ondisk_log_start":"94'8000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":202,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:20:44.142687+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:20:44.142687+0000","last_peered":"2023-10-20T10:20:44.142687+0000","last_clean":"2023-10-20T10:20:44.142687+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:20:44.142687+0000","last_undegraded":"2023-10-20T10:20:44.142687+0000","last_fullsized":"2023-10-20T10:20:44.142687+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"124'9141","reported_seq":14502,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-20T10:22:04.463362+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:22:04.463362+0000","last_peered":"2023-10-20T10:22:04.463362+0000","last_clean":"2023-10-20T10:22:04.463362+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:22:04.463362+0000","last_undegraded":"2023-10-20T10:22:04.463362+0000","last_fullsized":"2023-10-20T10:22:04.463362+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9141,"log_dups_size":0,"ondisk_log_size":9141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"124'466554","reported_seq":926067,"reported_epoch":124,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:22:03.964982+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:22:03.964982+0000","last_peered":"2023-10-20T10:22:03.964982+0000","last_clean":"2023-10-20T10:22:03.964982+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:22:03.964982+0000","last_undegraded":"2023-10-20T10:22:03.964982+0000","last_fullsized":"2023-10-20T10:22:03.964982+0000","mapping_epoch":16,"log_start":"122'456530","ondisk_log_start":"122'456530","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":989,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 140s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7015714816,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":462734,"num_read_kb":1821556,"num_write":462282,"num_write_kb":1849128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":296,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:20:43.489246+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:20:43.489246+0000","last_peered":"2023-10-20T10:20:43.489246+0000","last_clean":"2023-10-20T10:20:43.489246+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:20:43.489246+0000","last_undegraded":"2023-10-20T10:20:43.489246+0000","last_fullsized":"2023-10-20T10:20:43.489246+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"124'432694","reported_seq":858864,"reported_epoch":124,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:22:04.500710+0000","last_change":"2023-10-20T10:19:59.154102+0000","last_active":"2023-10-20T10:22:04.500710+0000","last_peered":"2023-10-20T10:22:04.500710+0000","last_clean":"2023-10-20T10:22:04.500710+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:22:04.500710+0000","last_undegraded":"2023-10-20T10:22:04.500710+0000","last_fullsized":"2023-10-20T10:22:04.500710+0000","mapping_epoch":16,"log_start":"122'422640","ondisk_log_start":"122'422640","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":1123,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 125s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6504067072,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":429174,"num_read_kb":1687280,"num_write":428548,"num_write_kb":1714192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"122'326","reported_seq":6297,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-20T10:22:03.934409+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:22:03.934409+0000","last_peered":"2023-10-20T10:22:03.934409+0000","last_clean":"2023-10-20T10:22:03.934409+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:22:03.934409+0000","last_undegraded":"2023-10-20T10:22:03.934409+0000","last_fullsized":"2023-10-20T10:22:03.934409+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"124'459804","reported_seq":912842,"reported_epoch":124,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:22:03.970424+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:22:03.970424+0000","last_peered":"2023-10-20T10:22:03.970424+0000","last_clean":"2023-10-20T10:22:03.970424+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:22:03.970424+0000","last_undegraded":"2023-10-20T10:22:03.970424+0000","last_fullsized":"2023-10-20T10:22:03.970424+0000","mapping_epoch":16,"log_start":"122'449735","ondisk_log_start":"122'449735","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":1001,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 139s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":6909792256,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":456276,"num_read_kb":1794376,"num_write":455500,"num_write_kb":1822000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"122'303","reported_seq":6147,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-20T10:22:03.938983+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:22:03.938983+0000","last_peered":"2023-10-20T10:22:03.938983+0000","last_clean":"2023-10-20T10:22:03.938983+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:22:03.938983+0000","last_undegraded":"2023-10-20T10:22:03.938983+0000","last_fullsized":"2023-10-20T10:22:03.938983+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"124'457092","reported_seq":908748,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-20T10:22:04.493919+0000","last_change":"2023-10-20T10:21:57.127844+0000","last_active":"2023-10-20T10:22:04.493919+0000","last_peered":"2023-10-20T10:22:04.493919+0000","last_clean":"2023-10-20T10:22:04.493919+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:22:04.493919+0000","last_undegraded":"2023-10-20T10:22:04.493919+0000","last_fullsized":"2023-10-20T10:22:04.493919+0000","mapping_epoch":16,"log_start":"122'447024","ondisk_log_start":"122'447024","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6874304531,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":454882,"num_read_kb":1789124,"num_write":453618,"num_write_kb":1814466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":201,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:20:44.142657+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:20:44.142657+0000","last_peered":"2023-10-20T10:20:44.142657+0000","last_clean":"2023-10-20T10:20:44.142657+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:20:44.142657+0000","last_undegraded":"2023-10-20T10:20:44.142657+0000","last_fullsized":"2023-10-20T10:20:44.142657+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":295,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-20T10:21:28.056281+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:21:28.056281+0000","last_peered":"2023-10-20T10:21:28.056281+0000","last_clean":"2023-10-20T10:21:28.056281+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:21:28.056281+0000","last_undegraded":"2023-10-20T10:21:28.056281+0000","last_fullsized":"2023-10-20T10:21:28.056281+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27303878675,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1803066,"num_read_kb":7092336,"num_write":1799948,"num_write_kb":7199786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14745079808,"data_stored":28127834150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40215,"ondisk_log_size":40215,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":29078,"ondisk_log_size":29078,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739193,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9066900,"kb_used_data":7200204,"kb_used_omap":1430,"kb_used_meta":1865257,"kb_avail":85304940,"statfs":{"total":96636764160,"available":87352258560,"internally_reserved":0,"allocated":7373008896,"data_stored":14063555334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1465289,"internal_metadata":1910023223},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,5,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:22:00 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.625,"15min":0.45100000000000001},"min":{"1min":0.246,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56699999999999995,"5min":8.2449999999999992,"15min":8.2449999999999992},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.57299999999999995,"15min":0.48999999999999999},"min":{"1min":0.20499999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.34699999999999998,"5min":6.8239999999999998,"15min":6.8239999999999998},"last":1.5760000000000001}]},{"osd":1,"last update":"Fri Oct 20 10:22:00 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.68000000000000005,"15min":0.71599999999999997},"min":{"1min":0.19900000000000001,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":1.6879999999999999,"5min":8.282,"15min":35.670000000000002},"last":1.6259999999999999},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.48299999999999998,"15min":0.64100000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.46600000000000003,"5min":4.899,"15min":34.948999999999998},"last":1.6539999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739193,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5004812,"kb_used_data":3530564,"kb_used_omap":676,"kb_used_meta":1473563,"kb_avail":89367028,"statfs":{"total":96636764160,"available":91511836672,"internally_reserved":0,"allocated":3615297536,"data_stored":6879506042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1508929099},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.51200000000000001,"15min":0.441},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.157},"max":{"1min":1.4330000000000001,"5min":4.0209999999999999,"15min":4.7039999999999997},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.40000000000000002,"15min":0.41899999999999998},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":1.5,"5min":5.8479999999999999,"15min":5.8479999999999999},"last":0.26300000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40600000000000003,"15min":0.38500000000000001},"min":{"1min":0.23899999999999999,"5min":0.19900000000000001,"15min":0.18099999999999999},"max":{"1min":0.378,"5min":3.657,"15min":4.8920000000000003},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.44600000000000001,"15min":0.42699999999999999},"min":{"1min":0.27500000000000002,"5min":0.23999999999999999,"15min":0.222},"max":{"1min":0.40400000000000003,"5min":3.6869999999999998,"15min":4.9279999999999999},"last":0.34699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739194,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5538268,"kb_used_data":3672724,"kb_used_omap":698,"kb_used_meta":1864837,"kb_avail":88833572,"statfs":{"total":96636764160,"available":90965577728,"internally_reserved":0,"allocated":3760869376,"data_stored":7186057195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":1909593674},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,9,6,11,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.46999999999999997,"15min":0.73699999999999999},"min":{"1min":0.19,"5min":0.17599999999999999,"15min":0.16700000000000001},"max":{"1min":0.50900000000000001,"5min":9.266,"15min":81.197000000000003},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.51900000000000002,"15min":0.68600000000000005},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.184},"max":{"1min":2.1400000000000001,"5min":7.2690000000000001,"15min":80.334999999999994},"last":0.23499999999999999}]},{"osd":2,"last update":"Fri Oct 20 10:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.621,"15min":0.53200000000000003},"min":{"1min":0.23999999999999999,"5min":0.23899999999999999,"15min":0.20899999999999999},"max":{"1min":0.56399999999999995,"5min":11.176,"15min":11.176},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.64900000000000002,"15min":0.61399999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.187},"max":{"1min":1.5049999999999999,"5min":11.093,"15min":11.093},"last":0.27400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3759239168,"data_stored":7185342464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3613347840,"data_stored":6878621715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7372492800,"data_stored":14063869971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:22:06.894 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:22:06.894 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:22:06.895 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:22:06.895 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:22:06.895 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-20T10:22:07.444 INFO:teuthology.orchestra.run.smithi150.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-20T10:22:07.479 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-20T10:22:07.948 INFO:teuthology.orchestra.run.smithi150.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-20T10:22:07.983 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-20T10:22:08.501 INFO:teuthology.orchestra.run.smithi150.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-20T10:22:08.538 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-20T10:22:09.088 INFO:teuthology.orchestra.run.smithi150.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-20T10:22:09.119 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:22:29.120 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:22:29.638 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:22:29.639 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:22:29.669 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2191,"stamp":"2023-10-20T10:22:29.232662+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27337413123,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1849792,"num_read_kb":7157117,"num_write":1808961,"num_write_kb":7234095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69518,"ondisk_log_size":69518,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19130880,"kb_used_data":14463528,"kb_used_omap":2807,"kb_used_meta":4664520,"kb_avail":263984640,"statfs":{"total":289910292480,"available":270320271360,"internally_reserved":0,"allocated":14810652672,"data_stored":28190595435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2874823,"internal_metadata":4776469049},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,2,10,8,12,5,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18718720,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4485,"num_read_kb":17844,"num_write":4551,"num_write_kb":18204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002000"},"pg_stats":[{"pgid":"2.7","version":"124'9211","reported_seq":13810,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-20T10:22:24.398611+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:22:24.398611+0000","last_peered":"2023-10-20T10:22:24.398611+0000","last_clean":"2023-10-20T10:22:24.398611+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:22:24.398611+0000","last_undegraded":"2023-10-20T10:22:24.398611+0000","last_fullsized":"2023-10-20T10:22:24.398611+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9211,"log_dups_size":0,"ondisk_log_size":9211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"124'18100","reported_seq":28291,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-20T10:22:24.398843+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:22:24.398843+0000","last_peered":"2023-10-20T10:22:24.398843+0000","last_clean":"2023-10-20T10:22:24.398843+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:22:24.398843+0000","last_undegraded":"2023-10-20T10:22:24.398843+0000","last_fullsized":"2023-10-20T10:22:24.398843+0000","mapping_epoch":13,"log_start":"94'8000","ondisk_log_start":"94'8000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":202,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:20:44.142687+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:20:44.142687+0000","last_peered":"2023-10-20T10:20:44.142687+0000","last_clean":"2023-10-20T10:20:44.142687+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:20:44.142687+0000","last_undegraded":"2023-10-20T10:20:44.142687+0000","last_fullsized":"2023-10-20T10:20:44.142687+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"124'9174","reported_seq":14547,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-20T10:22:24.398207+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:22:24.398207+0000","last_peered":"2023-10-20T10:22:24.398207+0000","last_clean":"2023-10-20T10:22:24.398207+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:22:24.398207+0000","last_undegraded":"2023-10-20T10:22:24.398207+0000","last_fullsized":"2023-10-20T10:22:24.398207+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9174,"log_dups_size":0,"ondisk_log_size":9174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"124'468946","reported_seq":930712,"reported_epoch":124,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:22:29.090353+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:22:29.090353+0000","last_peered":"2023-10-20T10:22:29.090353+0000","last_clean":"2023-10-20T10:22:29.090353+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:22:29.090353+0000","last_undegraded":"2023-10-20T10:22:29.090353+0000","last_fullsized":"2023-10-20T10:22:29.090353+0000","mapping_epoch":16,"log_start":"123'458930","ondisk_log_start":"123'458930","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":1161,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 165s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7024934912,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":464972,"num_read_kb":1830460,"num_write":464539,"num_write_kb":1858156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":296,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:20:43.489246+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:20:43.489246+0000","last_peered":"2023-10-20T10:20:43.489246+0000","last_clean":"2023-10-20T10:20:43.489246+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:20:43.489246+0000","last_undegraded":"2023-10-20T10:20:43.489246+0000","last_fullsized":"2023-10-20T10:20:43.489246+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"124'434505","reported_seq":862427,"reported_epoch":124,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:22:24.587195+0000","last_change":"2023-10-20T10:19:59.154102+0000","last_active":"2023-10-20T10:22:24.587195+0000","last_peered":"2023-10-20T10:22:24.587195+0000","last_clean":"2023-10-20T10:22:24.587195+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:22:24.587195+0000","last_undegraded":"2023-10-20T10:22:24.587195+0000","last_fullsized":"2023-10-20T10:22:24.587195+0000","mapping_epoch":16,"log_start":"123'424440","ondisk_log_start":"123'424440","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":1284,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 145s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6510989312,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":430898,"num_read_kb":1694140,"num_write":430232,"num_write_kb":1720928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"122'326","reported_seq":6309,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-20T10:22:19.279716+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:22:19.279716+0000","last_peered":"2023-10-20T10:22:19.279716+0000","last_clean":"2023-10-20T10:22:19.279716+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:22:19.279716+0000","last_undegraded":"2023-10-20T10:22:19.279716+0000","last_fullsized":"2023-10-20T10:22:19.279716+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"124'462211","reported_seq":917560,"reported_epoch":124,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:22:29.083877+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:22:29.083877+0000","last_peered":"2023-10-20T10:22:29.083877+0000","last_clean":"2023-10-20T10:22:29.083877+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:22:29.083877+0000","last_undegraded":"2023-10-20T10:22:29.083877+0000","last_fullsized":"2023-10-20T10:22:29.083877+0000","mapping_epoch":16,"log_start":"123'452135","ondisk_log_start":"123'452135","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":1180,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 164s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":6919225344,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":458570,"num_read_kb":1803508,"num_write":457773,"num_write_kb":1831092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"122'303","reported_seq":6159,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-20T10:22:19.286121+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:22:19.286121+0000","last_peered":"2023-10-20T10:22:19.286121+0000","last_clean":"2023-10-20T10:22:19.286121+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:22:19.286121+0000","last_undegraded":"2023-10-20T10:22:19.286121+0000","last_fullsized":"2023-10-20T10:22:19.286121+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"124'458877","reported_seq":912339,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-20T10:22:24.583171+0000","last_change":"2023-10-20T10:22:17.390869+0000","last_active":"2023-10-20T10:22:24.583171+0000","last_peered":"2023-10-20T10:22:24.583171+0000","last_clean":"2023-10-20T10:22:24.583171+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:22:24.583171+0000","last_undegraded":"2023-10-20T10:22:24.583171+0000","last_fullsized":"2023-10-20T10:22:24.583171+0000","mapping_epoch":16,"log_start":"123'448824","ondisk_log_start":"123'448824","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6881669139,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":456681,"num_read_kb":1796288,"num_write":455403,"num_write_kb":1821606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":201,"reported_epoch":123,"state":"active+clean","last_fresh":"2023-10-20T10:20:44.142657+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:20:44.142657+0000","last_peered":"2023-10-20T10:20:44.142657+0000","last_clean":"2023-10-20T10:20:44.142657+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:20:44.142657+0000","last_undegraded":"2023-10-20T10:20:44.142657+0000","last_fullsized":"2023-10-20T10:20:44.142657+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":295,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-20T10:21:28.056281+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:21:28.056281+0000","last_peered":"2023-10-20T10:21:28.056281+0000","last_clean":"2023-10-20T10:21:28.056281+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:21:28.056281+0000","last_undegraded":"2023-10-20T10:21:28.056281+0000","last_fullsized":"2023-10-20T10:21:28.056281+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27336818707,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1811121,"num_read_kb":7124396,"num_write":1807947,"num_write_kb":7231782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14806728704,"data_stored":28189483046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40210,"ondisk_log_size":40210,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":29209,"ondisk_log_size":29209,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739197,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8440864,"kb_used_data":7227992,"kb_used_omap":1432,"kb_used_meta":1211431,"kb_avail":85930976,"statfs":{"total":96636764160,"available":87993319424,"internally_reserved":0,"allocated":7401463808,"data_stored":14092010246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466972,"internal_metadata":1240505764},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:22:00 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.625,"15min":0.45100000000000001},"min":{"1min":0.246,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56699999999999995,"5min":8.2449999999999992,"15min":8.2449999999999992},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.57299999999999995,"15min":0.48999999999999999},"min":{"1min":0.20499999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.34699999999999998,"5min":6.8239999999999998,"15min":6.8239999999999998},"last":0.20799999999999999}]},{"osd":1,"last update":"Fri Oct 20 10:22:00 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.68000000000000005,"15min":0.71599999999999997},"min":{"1min":0.19900000000000001,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":1.6879999999999999,"5min":8.282,"15min":35.670000000000002},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.48299999999999998,"15min":0.64100000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.46600000000000003,"5min":4.899,"15min":34.948999999999998},"last":0.26300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739197,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5061232,"kb_used_data":3544488,"kb_used_omap":676,"kb_used_meta":1516059,"kb_avail":89310608,"statfs":{"total":96636764160,"available":91454062592,"internally_reserved":0,"allocated":3629555712,"data_stored":6893764218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1552445003},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,0,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.51200000000000001,"15min":0.441},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.157},"max":{"1min":1.4330000000000001,"5min":4.0209999999999999,"15min":4.7039999999999997},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.40000000000000002,"15min":0.41899999999999998},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":1.5,"5min":5.8479999999999999,"15min":5.8479999999999999},"last":0.218}]},{"osd":2,"last update":"Fri Oct 20 10:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40600000000000003,"15min":0.38500000000000001},"min":{"1min":0.23899999999999999,"5min":0.19900000000000001,"15min":0.18099999999999999},"max":{"1min":0.378,"5min":3.657,"15min":4.8920000000000003},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.44600000000000001,"15min":0.42699999999999999},"min":{"1min":0.27500000000000002,"5min":0.23999999999999999,"15min":0.222},"max":{"1min":0.40400000000000003,"5min":3.6869999999999998,"15min":4.9279999999999999},"last":0.50900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739199,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5628784,"kb_used_data":3691048,"kb_used_omap":698,"kb_used_meta":1937029,"kb_avail":88743056,"statfs":{"total":96636764160,"available":90872889344,"internally_reserved":0,"allocated":3779633152,"data_stored":7204820971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":1983518282},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,7,4,9,5,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.46999999999999997,"15min":0.73699999999999999},"min":{"1min":0.19,"5min":0.17599999999999999,"15min":0.16700000000000001},"max":{"1min":0.50900000000000001,"5min":9.266,"15min":81.197000000000003},"last":0.435},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.51900000000000002,"15min":0.68600000000000005},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.184},"max":{"1min":2.1400000000000001,"5min":7.2690000000000001,"15min":80.334999999999994},"last":0.27400000000000002}]},{"osd":2,"last update":"Fri Oct 20 10:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.621,"15min":0.53200000000000003},"min":{"1min":0.23999999999999999,"5min":0.23899999999999999,"15min":0.20899999999999999},"max":{"1min":0.56399999999999995,"5min":11.176,"15min":11.176},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.64900000000000002,"15min":0.61399999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.187},"max":{"1min":1.5049999999999999,"5min":11.093,"15min":11.093},"last":0.36299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3777798144,"data_stored":7203901440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3627565056,"data_stored":6892838931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7401365504,"data_stored":14092742675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:22:29.671 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:22:29.671 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:22:29.671 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:22:29.672 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:22:29.672 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:22:49.674 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:22:50.168 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:22:50.169 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:22:50.207 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2202,"stamp":"2023-10-20T10:22:49.236178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27366838787,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1857030,"num_read_kb":7185941,"num_write":1816100,"num_write_kb":7262651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69611,"ondisk_log_size":69611,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19314540,"kb_used_data":14520724,"kb_used_omap":2807,"kb_used_meta":4790984,"kb_avail":263800980,"statfs":{"total":289910292480,"available":270132203520,"internally_reserved":0,"allocated":14869221376,"data_stored":28249141228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2874823,"internal_metadata":4905968185},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,7,8,14,24,14,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18612224,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4554,"num_read_kb":18124,"num_write":4529,"num_write_kb":18116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.173252"},"pg_stats":[{"pgid":"2.7","version":"125'9247","reported_seq":13856,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:22:42.080781+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:22:42.080781+0000","last_peered":"2023-10-20T10:22:42.080781+0000","last_clean":"2023-10-20T10:22:42.080781+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:22:42.080781+0000","last_undegraded":"2023-10-20T10:22:42.080781+0000","last_fullsized":"2023-10-20T10:22:42.080781+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9247,"log_dups_size":0,"ondisk_log_size":9247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"125'18172","reported_seq":28382,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:22:42.104115+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:22:42.104115+0000","last_peered":"2023-10-20T10:22:42.104115+0000","last_clean":"2023-10-20T10:22:42.104115+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:22:42.104115+0000","last_undegraded":"2023-10-20T10:22:42.104115+0000","last_fullsized":"2023-10-20T10:22:42.104115+0000","mapping_epoch":13,"log_start":"96'8100","ondisk_log_start":"96'8100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":203,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-20T10:21:28.051176+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:21:28.051176+0000","last_peered":"2023-10-20T10:21:28.051176+0000","last_clean":"2023-10-20T10:21:28.051176+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:21:28.051176+0000","last_undegraded":"2023-10-20T10:21:28.051176+0000","last_fullsized":"2023-10-20T10:21:28.051176+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"125'9210","reported_seq":14593,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:22:42.083115+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:22:42.083115+0000","last_peered":"2023-10-20T10:22:42.083115+0000","last_clean":"2023-10-20T10:22:42.083115+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:22:42.083115+0000","last_undegraded":"2023-10-20T10:22:42.083115+0000","last_fullsized":"2023-10-20T10:22:42.083115+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9210,"log_dups_size":0,"ondisk_log_size":9210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"125'470889","reported_seq":934537,"reported_epoch":125,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:22:49.198934+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:22:49.198934+0000","last_peered":"2023-10-20T10:22:49.198934+0000","last_clean":"2023-10-20T10:22:49.198934+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:22:49.198934+0000","last_undegraded":"2023-10-20T10:22:49.198934+0000","last_fullsized":"2023-10-20T10:22:49.198934+0000","mapping_epoch":16,"log_start":"123'460830","ondisk_log_start":"123'460830","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":1287,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 185s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7032500224,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":466840,"num_read_kb":1837896,"num_write":466387,"num_write_kb":1865548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":298,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:22:38.062162+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:22:38.062162+0000","last_peered":"2023-10-20T10:22:38.062162+0000","last_clean":"2023-10-20T10:22:38.062162+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:22:38.062162+0000","last_undegraded":"2023-10-20T10:22:38.062162+0000","last_fullsized":"2023-10-20T10:22:38.062162+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"125'436274","reported_seq":865991,"reported_epoch":125,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:22:44.684262+0000","last_change":"2023-10-20T10:19:59.154102+0000","last_active":"2023-10-20T10:22:44.684262+0000","last_peered":"2023-10-20T10:22:44.684262+0000","last_clean":"2023-10-20T10:22:44.684262+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:22:44.684262+0000","last_undegraded":"2023-10-20T10:22:44.684262+0000","last_fullsized":"2023-10-20T10:22:44.684262+0000","mapping_epoch":16,"log_start":"123'426240","ondisk_log_start":"123'426240","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":1438,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 165s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6517788672,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":432676,"num_read_kb":1701224,"num_write":431888,"num_write_kb":1727552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"122'326","reported_seq":6327,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:22:39.758718+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:22:39.758718+0000","last_peered":"2023-10-20T10:22:39.758718+0000","last_clean":"2023-10-20T10:22:39.758718+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:22:39.758718+0000","last_undegraded":"2023-10-20T10:22:39.758718+0000","last_fullsized":"2023-10-20T10:22:39.758718+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"125'464110","reported_seq":921233,"reported_epoch":125,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:22:49.164628+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:22:49.164628+0000","last_peered":"2023-10-20T10:22:49.164628+0000","last_clean":"2023-10-20T10:22:49.164628+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:22:49.164628+0000","last_undegraded":"2023-10-20T10:22:49.164628+0000","last_fullsized":"2023-10-20T10:22:49.164628+0000","mapping_epoch":16,"log_start":"123'454035","ondisk_log_start":"123'454035","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":1320,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 184s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":6926704640,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":460334,"num_read_kb":1810528,"num_write":459570,"num_write_kb":1838280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"122'303","reported_seq":6176,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:22:39.759643+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:22:39.759643+0000","last_peered":"2023-10-20T10:22:39.759643+0000","last_clean":"2023-10-20T10:22:39.759643+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:22:39.759643+0000","last_undegraded":"2023-10-20T10:22:39.759643+0000","last_fullsized":"2023-10-20T10:22:39.759643+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"125'460715","reported_seq":916011,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:22:44.693469+0000","last_change":"2023-10-20T10:22:39.357340+0000","last_active":"2023-10-20T10:22:44.693469+0000","last_peered":"2023-10-20T10:22:44.693469+0000","last_clean":"2023-10-20T10:22:44.693469+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:22:44.693469+0000","last_undegraded":"2023-10-20T10:22:44.693469+0000","last_fullsized":"2023-10-20T10:22:44.693469+0000","mapping_epoch":16,"log_start":"123'450624","ondisk_log_start":"123'450624","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6889250835,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":458509,"num_read_kb":1803572,"num_write":457241,"num_write_kb":1828958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":202,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-20T10:21:28.050985+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:21:28.050985+0000","last_peered":"2023-10-20T10:21:28.050985+0000","last_clean":"2023-10-20T10:21:28.050985+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:21:28.050985+0000","last_undegraded":"2023-10-20T10:21:28.050985+0000","last_fullsized":"2023-10-20T10:21:28.050985+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":296,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-20T10:21:59.490834+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:21:59.490834+0000","last_peered":"2023-10-20T10:21:59.490834+0000","last_clean":"2023-10-20T10:21:59.490834+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:21:59.490834+0000","last_undegraded":"2023-10-20T10:21:59.490834+0000","last_fullsized":"2023-10-20T10:21:59.490834+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27366244371,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1818359,"num_read_kb":7153220,"num_write":1815086,"num_write_kb":7260338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14865326080,"data_stored":28248080422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40259,"ondisk_log_size":40259,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":29253,"ondisk_log_size":29253,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739201,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8559992,"kb_used_data":7256688,"kb_used_omap":1432,"kb_used_meta":1301863,"kb_avail":85811848,"statfs":{"total":96636764160,"available":87871332352,"internally_reserved":0,"allocated":7430848512,"data_stored":14121387313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466972,"internal_metadata":1333108132},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:22:00 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.625,"15min":0.45100000000000001},"min":{"1min":0.246,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56699999999999995,"5min":8.2449999999999992,"15min":8.2449999999999992},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.57299999999999995,"15min":0.48999999999999999},"min":{"1min":0.20499999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.34699999999999998,"5min":6.8239999999999998,"15min":6.8239999999999998},"last":0.32800000000000001}]},{"osd":1,"last update":"Fri Oct 20 10:22:00 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.68000000000000005,"15min":0.71599999999999997},"min":{"1min":0.19900000000000001,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":1.6879999999999999,"5min":8.282,"15min":35.670000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.48299999999999998,"15min":0.64100000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.46600000000000003,"5min":4.899,"15min":34.948999999999998},"last":0.36099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739201,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5093292,"kb_used_data":3558500,"kb_used_omap":676,"kb_used_meta":1534107,"kb_avail":89278548,"statfs":{"total":96636764160,"available":91421233152,"internally_reserved":0,"allocated":3643904000,"data_stored":6908104869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1570926155},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,3,4,11,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.51200000000000001,"15min":0.441},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.157},"max":{"1min":1.4330000000000001,"5min":4.0209999999999999,"15min":4.7039999999999997},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.40000000000000002,"15min":0.41899999999999998},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":1.5,"5min":5.8479999999999999,"15min":5.8479999999999999},"last":0.23100000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40600000000000003,"15min":0.38500000000000001},"min":{"1min":0.23899999999999999,"5min":0.19900000000000001,"15min":0.18099999999999999},"max":{"1min":0.378,"5min":3.657,"15min":4.8920000000000003},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.44600000000000001,"15min":0.42699999999999999},"min":{"1min":0.27500000000000002,"5min":0.23999999999999999,"15min":0.222},"max":{"1min":0.40400000000000003,"5min":3.6869999999999998,"15min":4.9279999999999999},"last":0.39700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739203,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5661256,"kb_used_data":3705536,"kb_used_omap":698,"kb_used_meta":1955013,"kb_avail":88710584,"statfs":{"total":96636764160,"available":90839638016,"internally_reserved":0,"allocated":3794468864,"data_stored":7219649046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":2001933898},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,7,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:22:46 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.41199999999999998,"15min":0.73999999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.16700000000000001},"max":{"1min":3.1509999999999998,"5min":3.6920000000000002,"15min":81.197000000000003},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.436,"15min":0.67700000000000005},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.184},"max":{"1min":0.44700000000000001,"5min":7.2690000000000001,"15min":80.334999999999994},"last":0.36899999999999999}]},{"osd":2,"last update":"Fri Oct 20 10:22:46 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.53400000000000003,"15min":0.51900000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20899999999999999},"max":{"1min":0.47899999999999998,"5min":11.176,"15min":11.176},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.56200000000000006,"15min":0.60099999999999998},"min":{"1min":0.21199999999999999,"5min":0.20000000000000001,"15min":0.187},"max":{"1min":0.44600000000000001,"5min":11.093,"15min":11.093},"last":0.35099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3792719872,"data_stored":7218823168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3641872384,"data_stored":6907146259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7430733824,"data_stored":14122110995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:22:50.209 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:22:50.210 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:22:50.210 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:22:50.211 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:22:50.211 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:23:10.213 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:23:10.736 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:23:10.737 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:23:10.773 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2212,"stamp":"2023-10-20T10:23:09.239499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27391656451,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1863176,"num_read_kb":7210449,"num_write":1822126,"num_write_kb":7286755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69651,"ondisk_log_size":69651,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19552660,"kb_used_data":14572796,"kb_used_omap":2807,"kb_used_meta":4977032,"kb_avail":263562860,"statfs":{"total":289910292480,"available":269888368640,"internally_reserved":0,"allocated":14922543104,"data_stored":28302462956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2874823,"internal_metadata":5096481337},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,7,3,13,8,11,18],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14090240,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3547,"num_read_kb":14148,"num_write":3418,"num_write_kb":13672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002008"},"pg_stats":[{"pgid":"2.7","version":"125'9304","reported_seq":13925,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:23:03.799914+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:23:03.799914+0000","last_peered":"2023-10-20T10:23:03.799914+0000","last_clean":"2023-10-20T10:23:03.799914+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:23:03.799914+0000","last_undegraded":"2023-10-20T10:23:03.799914+0000","last_fullsized":"2023-10-20T10:23:03.799914+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9304,"log_dups_size":0,"ondisk_log_size":9304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"125'18286","reported_seq":28520,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:23:03.814838+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:23:03.814838+0000","last_peered":"2023-10-20T10:23:03.814838+0000","last_clean":"2023-10-20T10:23:03.814838+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:23:03.814838+0000","last_undegraded":"2023-10-20T10:23:03.814838+0000","last_fullsized":"2023-10-20T10:23:03.814838+0000","mapping_epoch":13,"log_start":"96'8200","ondisk_log_start":"96'8200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":204,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:22:38.074068+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:22:38.074068+0000","last_peered":"2023-10-20T10:22:38.074068+0000","last_clean":"2023-10-20T10:22:38.074068+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:22:38.074068+0000","last_undegraded":"2023-10-20T10:22:38.074068+0000","last_fullsized":"2023-10-20T10:22:38.074068+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"125'9267","reported_seq":14663,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:23:03.786933+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:23:03.786933+0000","last_peered":"2023-10-20T10:23:03.786933+0000","last_clean":"2023-10-20T10:23:03.786933+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:23:03.786933+0000","last_undegraded":"2023-10-20T10:23:03.786933+0000","last_fullsized":"2023-10-20T10:23:03.786933+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9267,"log_dups_size":0,"ondisk_log_size":9267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"125'472316","reported_seq":937315,"reported_epoch":125,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:23:04.259885+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:23:04.259885+0000","last_peered":"2023-10-20T10:23:04.259885+0000","last_clean":"2023-10-20T10:23:04.259885+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:23:04.259885+0000","last_undegraded":"2023-10-20T10:23:04.259885+0000","last_fullsized":"2023-10-20T10:23:04.259885+0000","mapping_epoch":16,"log_start":"123'462230","ondisk_log_start":"123'462230","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":1440,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 200s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7037857792,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":468180,"num_read_kb":1843236,"num_write":467696,"num_write_kb":1870784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":298,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:22:38.062162+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:22:38.062162+0000","last_peered":"2023-10-20T10:22:38.062162+0000","last_clean":"2023-10-20T10:22:38.062162+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:22:38.062162+0000","last_undegraded":"2023-10-20T10:22:38.062162+0000","last_fullsized":"2023-10-20T10:22:38.062162+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"125'438053","reported_seq":869498,"reported_epoch":125,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:23:04.765622+0000","last_change":"2023-10-20T10:19:59.154102+0000","last_active":"2023-10-20T10:23:04.765622+0000","last_peered":"2023-10-20T10:23:04.765622+0000","last_clean":"2023-10-20T10:23:04.765622+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:23:04.765622+0000","last_undegraded":"2023-10-20T10:23:04.765622+0000","last_fullsized":"2023-10-20T10:23:04.765622+0000","mapping_epoch":16,"log_start":"123'428040","ondisk_log_start":"123'428040","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":1658,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 185s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6524379136,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":434385,"num_read_kb":1708044,"num_write":433494,"num_write_kb":1733976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"122'326","reported_seq":6343,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:23:00.238731+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:23:00.238731+0000","last_peered":"2023-10-20T10:23:00.238731+0000","last_clean":"2023-10-20T10:23:00.238731+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:23:00.238731+0000","last_undegraded":"2023-10-20T10:23:00.238731+0000","last_fullsized":"2023-10-20T10:23:00.238731+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"125'465536","reported_seq":924011,"reported_epoch":125,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:23:04.241357+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:23:04.241357+0000","last_peered":"2023-10-20T10:23:04.241357+0000","last_clean":"2023-10-20T10:23:04.241357+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:23:04.241357+0000","last_undegraded":"2023-10-20T10:23:04.241357+0000","last_fullsized":"2023-10-20T10:23:04.241357+0000","mapping_epoch":16,"log_start":"123'455535","ondisk_log_start":"123'455535","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":1446,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 199s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":6932213760,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":461673,"num_read_kb":1815872,"num_write":460901,"num_write_kb":1843604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"122'303","reported_seq":6192,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:23:00.249507+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:23:00.249507+0000","last_peered":"2023-10-20T10:23:00.249507+0000","last_clean":"2023-10-20T10:23:00.249507+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:23:00.249507+0000","last_undegraded":"2023-10-20T10:23:00.249507+0000","last_fullsized":"2023-10-20T10:23:00.249507+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"125'462495","reported_seq":919555,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:23:04.763946+0000","last_change":"2023-10-20T10:23:04.451300+0000","last_active":"2023-10-20T10:23:04.763946+0000","last_peered":"2023-10-20T10:23:04.763946+0000","last_clean":"2023-10-20T10:23:04.763946+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:23:04.763946+0000","last_undegraded":"2023-10-20T10:23:04.763946+0000","last_fullsized":"2023-10-20T10:23:04.763946+0000","mapping_epoch":16,"log_start":"123'452424","ondisk_log_start":"123'452424","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6896611347,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":460267,"num_read_kb":1810576,"num_write":459021,"num_write_kb":1836078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":203,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:22:38.074067+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:22:38.074067+0000","last_peered":"2023-10-20T10:22:38.074067+0000","last_clean":"2023-10-20T10:22:38.074067+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:22:38.074067+0000","last_undegraded":"2023-10-20T10:22:38.074067+0000","last_fullsized":"2023-10-20T10:22:38.074067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":296,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-20T10:21:59.490834+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:21:59.490834+0000","last_peered":"2023-10-20T10:21:59.490834+0000","last_clean":"2023-10-20T10:21:59.490834+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:21:59.490834+0000","last_undegraded":"2023-10-20T10:21:59.490834+0000","last_fullsized":"2023-10-20T10:21:59.490834+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27391062035,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1824505,"num_read_kb":7177728,"num_write":1821112,"num_write_kb":7284442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14918311936,"data_stored":28301066278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40171,"ondisk_log_size":40171,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":29381,"ondisk_log_size":29381,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739205,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8692068,"kb_used_data":7284636,"kb_used_omap":1432,"kb_used_meta":1405991,"kb_avail":85679772,"statfs":{"total":96636764160,"available":87736086528,"internally_reserved":0,"allocated":7459467264,"data_stored":14150006065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466972,"internal_metadata":1439735204},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.53600000000000003,"15min":0.45100000000000001},"min":{"1min":0.17000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":1.806,"5min":6.8419999999999996,"15min":8.2449999999999992},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.55600000000000005,"15min":0.496},"min":{"1min":0.20799999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.6419999999999999,"5min":6.8239999999999998,"15min":6.8239999999999998},"last":0.28799999999999998}]},{"osd":1,"last update":"Fri Oct 20 10:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.60999999999999999,"15min":0.73599999999999999},"min":{"1min":0.19700000000000001,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":3.5150000000000001,"5min":8.282,"15min":35.670000000000002},"last":0.254},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.47899999999999998,"15min":0.63900000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.214},"max":{"1min":2.7090000000000001,"5min":4.899,"15min":34.948999999999998},"last":0.318}]}]},{"osd":1,"up_from":8,"seq":34359739205,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5138668,"kb_used_data":3571940,"kb_used_omap":676,"kb_used_meta":1566043,"kb_avail":89233172,"statfs":{"total":96636764160,"available":91374768128,"internally_reserved":0,"allocated":3657666560,"data_stored":6921867429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1603628619},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:22:49 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.501,"15min":0.45600000000000002},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.157},"max":{"1min":3.8650000000000002,"5min":4.0209999999999999,"15min":4.7039999999999997},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.38400000000000001,"15min":0.433},"min":{"1min":0.20300000000000001,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":4.7110000000000003,"5min":4.7110000000000003,"15min":5.8479999999999999},"last":0.28599999999999998}]},{"osd":2,"last update":"Fri Oct 20 10:22:49 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.36699999999999999,"15min":0.39700000000000002},"min":{"1min":0.217,"5min":0.19900000000000001,"15min":0.18099999999999999},"max":{"1min":3.8250000000000002,"5min":3.8250000000000002,"15min":4.8920000000000003},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.41199999999999998,"15min":0.442},"min":{"1min":0.28699999999999998,"5min":0.24099999999999999,"15min":0.23799999999999999},"max":{"1min":3.9020000000000001,"5min":3.9020000000000001,"15min":4.9279999999999999},"last":0.42499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739206,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5721924,"kb_used_data":3716220,"kb_used_omap":698,"kb_used_meta":2004997,"kb_avail":88649916,"statfs":{"total":96636764160,"available":90777513984,"internally_reserved":0,"allocated":3805409280,"data_stored":7230589462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":2053117514},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,2,5,6,11,18],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:22:46 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.41199999999999998,"15min":0.73999999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.16700000000000001},"max":{"1min":3.1509999999999998,"5min":3.6920000000000002,"15min":81.197000000000003},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.436,"15min":0.67700000000000005},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.184},"max":{"1min":0.44700000000000001,"5min":7.2690000000000001,"15min":80.334999999999994},"last":0.28799999999999998}]},{"osd":2,"last update":"Fri Oct 20 10:22:46 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.53400000000000003,"15min":0.51900000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20899999999999999},"max":{"1min":0.47899999999999998,"5min":11.176,"15min":11.176},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.56200000000000006,"15min":0.60099999999999998},"min":{"1min":0.21199999999999999,"5min":0.20000000000000001,"15min":0.187},"max":{"1min":0.44600000000000001,"5min":11.093,"15min":11.093},"last":0.36699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3803537408,"data_stored":7229640704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3655753728,"data_stored":6921027603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7459020800,"data_stored":14150397971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:23:10.775 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:23:10.776 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:23:10.776 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:23:10.776 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:23:10.777 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:23:29.767 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:23:29.766+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T10:23:29.768 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:23:29.766+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:23:29.769 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:23:29.766+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:23:30.778 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:23:31.262 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:23:31.263 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:23:31.306 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2223,"stamp":"2023-10-20T10:23:31.243337+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27428065795,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1872181,"num_read_kb":7246345,"num_write":1830994,"num_write_kb":7322227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69815,"ondisk_log_size":69815,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19852588,"kb_used_data":14643668,"kb_used_omap":2807,"kb_used_meta":5206088,"kb_avail":263262932,"statfs":{"total":289910292480,"available":269581242368,"internally_reserved":0,"allocated":14995116032,"data_stored":28375035884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2874823,"internal_metadata":5331034681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,5,44,21,15,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21770240,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5390,"num_read_kb":21508,"num_write":5312,"num_write_kb":21248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002058"},"pg_stats":[{"pgid":"2.7","version":"125'9357","reported_seq":13994,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:23:29.784231+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:23:29.784231+0000","last_peered":"2023-10-20T10:23:29.784231+0000","last_clean":"2023-10-20T10:23:29.784231+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:23:29.784231+0000","last_undegraded":"2023-10-20T10:23:29.784231+0000","last_fullsized":"2023-10-20T10:23:29.784231+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9357,"log_dups_size":0,"ondisk_log_size":9357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"125'18389","reported_seq":28654,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:23:29.786175+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:23:29.786175+0000","last_peered":"2023-10-20T10:23:29.786175+0000","last_clean":"2023-10-20T10:23:29.786175+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:23:29.786175+0000","last_undegraded":"2023-10-20T10:23:29.786175+0000","last_fullsized":"2023-10-20T10:23:29.786175+0000","mapping_epoch":13,"log_start":"96'8300","ondisk_log_start":"96'8300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":204,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:22:38.074068+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:22:38.074068+0000","last_peered":"2023-10-20T10:22:38.074068+0000","last_clean":"2023-10-20T10:22:38.074068+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:22:38.074068+0000","last_undegraded":"2023-10-20T10:22:38.074068+0000","last_fullsized":"2023-10-20T10:22:38.074068+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"125'9319","reported_seq":14730,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:23:29.763728+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:23:29.763728+0000","last_peered":"2023-10-20T10:23:29.763728+0000","last_clean":"2023-10-20T10:23:29.763728+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:23:29.763728+0000","last_undegraded":"2023-10-20T10:23:29.763728+0000","last_fullsized":"2023-10-20T10:23:29.763728+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9319,"log_dups_size":0,"ondisk_log_size":9319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"125'474728","reported_seq":942047,"reported_epoch":125,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:23:29.380747+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:23:29.380747+0000","last_peered":"2023-10-20T10:23:29.380747+0000","last_clean":"2023-10-20T10:23:29.380747+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:23:29.380747+0000","last_undegraded":"2023-10-20T10:23:29.380747+0000","last_fullsized":"2023-10-20T10:23:29.380747+0000","mapping_epoch":16,"log_start":"124'464631","ondisk_log_start":"124'464631","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":1607,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 225s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7047245824,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":470485,"num_read_kb":1852416,"num_write":469973,"num_write_kb":1879892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":298,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:22:38.062162+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:22:38.062162+0000","last_peered":"2023-10-20T10:22:38.062162+0000","last_clean":"2023-10-20T10:22:38.062162+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:22:38.062162+0000","last_undegraded":"2023-10-20T10:22:38.062162+0000","last_fullsized":"2023-10-20T10:22:38.062162+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"125'440332","reported_seq":873934,"reported_epoch":125,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:23:29.873866+0000","last_change":"2023-10-20T10:19:59.154102+0000","last_active":"2023-10-20T10:23:29.873866+0000","last_peered":"2023-10-20T10:23:29.873866+0000","last_clean":"2023-10-20T10:23:29.873866+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:23:29.873866+0000","last_undegraded":"2023-10-20T10:23:29.873866+0000","last_fullsized":"2023-10-20T10:23:29.873866+0000","mapping_epoch":16,"log_start":"124'430240","ondisk_log_start":"124'430240","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":1893,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 210s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6532960256,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":436526,"num_read_kb":1716588,"num_write":435589,"num_write_kb":1742356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"122'326","reported_seq":6363,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:23:25.838616+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:23:25.838616+0000","last_peered":"2023-10-20T10:23:25.838616+0000","last_clean":"2023-10-20T10:23:25.838616+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:23:25.838616+0000","last_undegraded":"2023-10-20T10:23:25.838616+0000","last_fullsized":"2023-10-20T10:23:25.838616+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"125'467942","reported_seq":928708,"reported_epoch":125,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:23:29.376398+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:23:29.376398+0000","last_peered":"2023-10-20T10:23:29.376398+0000","last_clean":"2023-10-20T10:23:29.376398+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:23:29.376398+0000","last_undegraded":"2023-10-20T10:23:29.376398+0000","last_fullsized":"2023-10-20T10:23:29.376398+0000","mapping_epoch":16,"log_start":"124'457935","ondisk_log_start":"124'457935","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":1662,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 224s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":6941405184,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":463946,"num_read_kb":1824928,"num_write":463137,"num_write_kb":1852548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"122'303","reported_seq":6212,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:23:25.839045+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:23:25.839045+0000","last_peered":"2023-10-20T10:23:25.839045+0000","last_clean":"2023-10-20T10:23:25.839045+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:23:25.839045+0000","last_undegraded":"2023-10-20T10:23:25.839045+0000","last_fullsized":"2023-10-20T10:23:25.839045+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"125'464755","reported_seq":924108,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:23:29.873266+0000","last_change":"2023-10-20T10:23:28.706205+0000","last_active":"2023-10-20T10:23:29.873266+0000","last_peered":"2023-10-20T10:23:29.873266+0000","last_clean":"2023-10-20T10:23:29.873266+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:23:29.873266+0000","last_undegraded":"2023-10-20T10:23:29.873266+0000","last_fullsized":"2023-10-20T10:23:29.873266+0000","mapping_epoch":16,"log_start":"124'454724","ondisk_log_start":"124'454724","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6905860115,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":462553,"num_read_kb":1819692,"num_write":461281,"num_write_kb":1845118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":203,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:22:38.074067+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:22:38.074067+0000","last_peered":"2023-10-20T10:22:38.074067+0000","last_clean":"2023-10-20T10:22:38.074067+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:22:38.074067+0000","last_undegraded":"2023-10-20T10:22:38.074067+0000","last_fullsized":"2023-10-20T10:22:38.074067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":296,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-20T10:21:59.490834+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:21:59.490834+0000","last_peered":"2023-10-20T10:21:59.490834+0000","last_clean":"2023-10-20T10:21:59.490834+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:21:59.490834+0000","last_undegraded":"2023-10-20T10:21:59.490834+0000","last_fullsized":"2023-10-20T10:21:59.490834+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27427471379,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1833510,"num_read_kb":7213624,"num_write":1829980,"num_write_kb":7319914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14990929920,"data_stored":28373684262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40227,"ondisk_log_size":40227,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":29489,"ondisk_log_size":29489,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739210,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8839852,"kb_used_data":7319972,"kb_used_omap":1432,"kb_used_meta":1518439,"kb_avail":85531988,"statfs":{"total":96636764160,"available":87584755712,"internally_reserved":0,"allocated":7495651328,"data_stored":14186190129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466972,"internal_metadata":1554881956},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.53600000000000003,"15min":0.45100000000000001},"min":{"1min":0.17000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":1.806,"5min":6.8419999999999996,"15min":8.2449999999999992},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.55600000000000005,"15min":0.496},"min":{"1min":0.20799999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.6419999999999999,"5min":6.8239999999999998,"15min":6.8239999999999998},"last":0.35499999999999998}]},{"osd":1,"last update":"Fri Oct 20 10:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.60999999999999999,"15min":0.73599999999999999},"min":{"1min":0.19700000000000001,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":3.5150000000000001,"5min":8.282,"15min":35.670000000000002},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.47899999999999998,"15min":0.63900000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.214},"max":{"1min":2.7090000000000001,"5min":4.899,"15min":34.948999999999998},"last":0.29699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739210,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5232556,"kb_used_data":3589476,"kb_used_omap":676,"kb_used_meta":1642395,"kb_avail":89139284,"statfs":{"total":96636764160,"available":91278626816,"internally_reserved":0,"allocated":3675623424,"data_stored":6939824293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1681813067},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,40,7,7,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:22:49 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.501,"15min":0.45600000000000002},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.157},"max":{"1min":3.8650000000000002,"5min":4.0209999999999999,"15min":4.7039999999999997},"last":4.7510000000000003},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.38400000000000001,"15min":0.433},"min":{"1min":0.20300000000000001,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":4.7110000000000003,"5min":4.7110000000000003,"15min":5.8479999999999999},"last":5.4640000000000004}]},{"osd":2,"last update":"Fri Oct 20 10:22:49 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.36699999999999999,"15min":0.39700000000000002},"min":{"1min":0.217,"5min":0.19900000000000001,"15min":0.18099999999999999},"max":{"1min":3.8250000000000002,"5min":3.8250000000000002,"15min":4.8920000000000003},"last":4.7859999999999996},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.41199999999999998,"15min":0.442},"min":{"1min":0.28699999999999998,"5min":0.24099999999999999,"15min":0.23799999999999999},"max":{"1min":3.9020000000000001,"5min":3.9020000000000001,"15min":4.9279999999999999},"last":4.8109999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739211,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5780180,"kb_used_data":3734220,"kb_used_omap":698,"kb_used_meta":2045253,"kb_avail":88591660,"statfs":{"total":96636764160,"available":90717859840,"internally_reserved":0,"allocated":3823841280,"data_stored":7249021462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":2094339658},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,12,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:22:46 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.41199999999999998,"15min":0.73999999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.16700000000000001},"max":{"1min":3.1509999999999998,"5min":3.6920000000000002,"15min":81.197000000000003},"last":0.377},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.436,"15min":0.67700000000000005},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.184},"max":{"1min":0.44700000000000001,"5min":7.2690000000000001,"15min":80.334999999999994},"last":0.23200000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:22:46 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.53400000000000003,"15min":0.51900000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20899999999999999},"max":{"1min":0.47899999999999998,"5min":11.176,"15min":11.176},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.56200000000000006,"15min":0.60099999999999998},"min":{"1min":0.21199999999999999,"5min":0.20000000000000001,"15min":0.187},"max":{"1min":0.44600000000000001,"5min":11.093,"15min":11.093},"last":0.20599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3822022656,"data_stored":7248125952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3673595904,"data_stored":6938869779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7495311360,"data_stored":14186688531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:23:31.308 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:23:31.308 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:23:31.309 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:23:31.309 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:23:31.310 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:23:45.246 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:23:45.241+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:23:45.262 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:23:45.257+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:23:51.311 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:23:51.974 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:23:51.975 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:23:52.005 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2234,"stamp":"2023-10-20T10:23:51.248606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27457384963,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1879417,"num_read_kb":7275181,"num_write":1838130,"num_write_kb":7350771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69702,"ondisk_log_size":69702,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20091892,"kb_used_data":14700700,"kb_used_omap":2807,"kb_used_meta":5388360,"kb_avail":263023628,"statfs":{"total":289910292480,"available":269336195072,"internally_reserved":0,"allocated":15053516800,"data_stored":28433413612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2874823,"internal_metadata":5517681209},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,5,8,11,13,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14700544,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3690,"num_read_kb":14708,"num_write":3589,"num_write_kb":14356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003541"},"pg_stats":[{"pgid":"2.7","version":"126'9388","reported_seq":14038,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:23:47.979967+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:23:47.979967+0000","last_peered":"2023-10-20T10:23:47.979967+0000","last_clean":"2023-10-20T10:23:47.979967+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:23:47.979967+0000","last_undegraded":"2023-10-20T10:23:47.979967+0000","last_fullsized":"2023-10-20T10:23:47.979967+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9388,"log_dups_size":0,"ondisk_log_size":9388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"126'18450","reported_seq":28740,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:23:48.007686+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:23:48.007686+0000","last_peered":"2023-10-20T10:23:48.007686+0000","last_clean":"2023-10-20T10:23:48.007686+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:23:48.007686+0000","last_undegraded":"2023-10-20T10:23:48.007686+0000","last_fullsized":"2023-10-20T10:23:48.007686+0000","mapping_epoch":13,"log_start":"96'8400","ondisk_log_start":"96'8400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":205,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:22:49.708360+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:22:49.708360+0000","last_peered":"2023-10-20T10:22:49.708360+0000","last_clean":"2023-10-20T10:22:49.708360+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:22:49.708360+0000","last_undegraded":"2023-10-20T10:22:49.708360+0000","last_fullsized":"2023-10-20T10:22:49.708360+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"126'9350","reported_seq":14774,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:23:48.006539+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:23:48.006539+0000","last_peered":"2023-10-20T10:23:48.006539+0000","last_clean":"2023-10-20T10:23:48.006539+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:23:48.006539+0000","last_undegraded":"2023-10-20T10:23:48.006539+0000","last_fullsized":"2023-10-20T10:23:48.006539+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9350,"log_dups_size":0,"ondisk_log_size":9350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"126'476652","reported_seq":945845,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:23:49.450243+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:23:49.450243+0000","last_peered":"2023-10-20T10:23:49.450243+0000","last_clean":"2023-10-20T10:23:49.450243+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:23:49.450243+0000","last_undegraded":"2023-10-20T10:23:49.450243+0000","last_fullsized":"2023-10-20T10:23:49.450243+0000","mapping_epoch":16,"log_start":"124'466631","ondisk_log_start":"124'466631","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":1742,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 245s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7054741504,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":472347,"num_read_kb":1859844,"num_write":471793,"num_write_kb":1887172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":299,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:22:49.194947+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:22:49.194947+0000","last_peered":"2023-10-20T10:22:49.194947+0000","last_clean":"2023-10-20T10:22:49.194947+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:22:49.194947+0000","last_undegraded":"2023-10-20T10:22:49.194947+0000","last_fullsized":"2023-10-20T10:22:49.194947+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"126'442150","reported_seq":877486,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:23:49.989135+0000","last_change":"2023-10-20T10:19:59.154102+0000","last_active":"2023-10-20T10:23:49.989135+0000","last_peered":"2023-10-20T10:23:49.989135+0000","last_clean":"2023-10-20T10:23:49.989135+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:23:49.989135+0000","last_undegraded":"2023-10-20T10:23:49.989135+0000","last_fullsized":"2023-10-20T10:23:49.989135+0000","mapping_epoch":16,"log_start":"124'432141","ondisk_log_start":"124'432141","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":2047,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 230s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6539939840,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":438246,"num_read_kb":1723440,"num_write":437290,"num_write_kb":1749160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"126'330","reported_seq":6384,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:23:46.365153+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:23:46.365153+0000","last_peered":"2023-10-20T10:23:46.365153+0000","last_clean":"2023-10-20T10:23:46.365153+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:23:46.365153+0000","last_undegraded":"2023-10-20T10:23:46.365153+0000","last_fullsized":"2023-10-20T10:23:46.365153+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"126'469873","reported_seq":932476,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:23:49.449164+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:23:49.449164+0000","last_peered":"2023-10-20T10:23:49.449164+0000","last_clean":"2023-10-20T10:23:49.449164+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:23:49.449164+0000","last_undegraded":"2023-10-20T10:23:49.449164+0000","last_fullsized":"2023-10-20T10:23:49.449164+0000","mapping_epoch":16,"log_start":"124'459835","ondisk_log_start":"124'459835","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":1794,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 244s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":6948937728,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":465771,"num_read_kb":1832200,"num_write":464968,"num_write_kb":1859872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"126'307","reported_seq":6233,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:23:46.372453+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:23:46.372453+0000","last_peered":"2023-10-20T10:23:46.372453+0000","last_clean":"2023-10-20T10:23:46.372453+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:23:46.372453+0000","last_undegraded":"2023-10-20T10:23:46.372453+0000","last_fullsized":"2023-10-20T10:23:46.372453+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"126'466539","reported_seq":927724,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:23:49.982575+0000","last_change":"2023-10-20T10:23:49.934972+0000","last_active":"2023-10-20T10:23:49.982575+0000","last_peered":"2023-10-20T10:23:49.982575+0000","last_clean":"2023-10-20T10:23:49.982575+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:23:49.982575+0000","last_undegraded":"2023-10-20T10:23:49.982575+0000","last_fullsized":"2023-10-20T10:23:49.982575+0000","mapping_epoch":16,"log_start":"124'456524","ondisk_log_start":"124'456524","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6913171475,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":464382,"num_read_kb":1826976,"num_write":463065,"num_write_kb":1852254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":204,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:22:49.708317+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:22:49.708317+0000","last_peered":"2023-10-20T10:22:49.708317+0000","last_clean":"2023-10-20T10:22:49.708317+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:22:49.708317+0000","last_undegraded":"2023-10-20T10:22:49.708317+0000","last_fullsized":"2023-10-20T10:22:49.708317+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":297,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:22:38.077906+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:22:38.077906+0000","last_peered":"2023-10-20T10:22:38.077906+0000","last_clean":"2023-10-20T10:22:38.077906+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:22:38.077906+0000","last_undegraded":"2023-10-20T10:22:38.077906+0000","last_fullsized":"2023-10-20T10:22:38.077906+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27456790547,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1840746,"num_read_kb":7242460,"num_write":1837116,"num_write_kb":7348458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15049392128,"data_stored":28432146470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40083,"ondisk_log_size":40083,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":29520,"ondisk_log_size":29520,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739214,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8964384,"kb_used_data":7348312,"kb_used_omap":1432,"kb_used_meta":1614631,"kb_avail":85407456,"statfs":{"total":96636764160,"available":87457234944,"internally_reserved":0,"allocated":7524671488,"data_stored":14215202609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466972,"internal_metadata":1653382564},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.53600000000000003,"15min":0.45100000000000001},"min":{"1min":0.17000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":1.806,"5min":6.8419999999999996,"15min":8.2449999999999992},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.55600000000000005,"15min":0.496},"min":{"1min":0.20799999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.6419999999999999,"5min":6.8239999999999998,"15min":6.8239999999999998},"last":0.27000000000000002}]},{"osd":1,"last update":"Fri Oct 20 10:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.60999999999999999,"15min":0.73599999999999999},"min":{"1min":0.19700000000000001,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":3.5150000000000001,"5min":8.282,"15min":35.670000000000002},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.47899999999999998,"15min":0.63900000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.214},"max":{"1min":2.7090000000000001,"5min":4.899,"15min":34.948999999999998},"last":0.32200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739214,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5260448,"kb_used_data":3603480,"kb_used_omap":676,"kb_used_meta":1656283,"kb_avail":89111392,"statfs":{"total":96636764160,"available":91250065408,"internally_reserved":0,"allocated":3689963520,"data_stored":6954156709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1696034379},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,1,4,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:22:49 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.501,"15min":0.45600000000000002},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.157},"max":{"1min":3.8650000000000002,"5min":4.0209999999999999,"15min":4.7039999999999997},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.38400000000000001,"15min":0.433},"min":{"1min":0.20300000000000001,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":4.7110000000000003,"5min":4.7110000000000003,"15min":5.8479999999999999},"last":0.246}]},{"osd":2,"last update":"Fri Oct 20 10:22:49 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.36699999999999999,"15min":0.39700000000000002},"min":{"1min":0.217,"5min":0.19900000000000001,"15min":0.18099999999999999},"max":{"1min":3.8250000000000002,"5min":3.8250000000000002,"15min":4.8920000000000003},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.41199999999999998,"15min":0.442},"min":{"1min":0.28699999999999998,"5min":0.24099999999999999,"15min":0.23799999999999999},"max":{"1min":3.9020000000000001,"5min":3.9020000000000001,"15min":4.9279999999999999},"last":0.312}]}]},{"osd":0,"up_from":8,"seq":34359739215,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5867060,"kb_used_data":3748908,"kb_used_omap":698,"kb_used_meta":2117445,"kb_avail":88504780,"statfs":{"total":96636764160,"available":90628894720,"internally_reserved":0,"allocated":3838881792,"data_stored":7264054294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":2168264266},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,2,6,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:23:46 2023","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":0.64900000000000002,"15min":0.82499999999999996},"min":{"1min":0.17899999999999999,"5min":0.17799999999999999,"15min":0.16700000000000001},"max":{"1min":8.0869999999999997,"5min":8.0869999999999997,"15min":81.197000000000003},"last":4.4260000000000002},{"interface":"front","average":{"1min":1.052,"5min":0.59199999999999997,"15min":0.72399999999999998},"min":{"1min":0.21299999999999999,"5min":0.20000000000000001,"15min":0.184},"max":{"1min":6.875,"5min":7.2690000000000001,"15min":80.334999999999994},"last":0.24199999999999999}]},{"osd":2,"last update":"Fri Oct 20 10:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.40699999999999997,"15min":0.51900000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.53100000000000003,"5min":3.653,"15min":11.176},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.40200000000000002,"15min":0.58899999999999997},"min":{"1min":0.20599999999999999,"5min":0.20000000000000001,"15min":0.187},"max":{"1min":1.093,"5min":3.4209999999999998,"15min":11.093},"last":0.45700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3836973056,"data_stored":7263076352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3687866368,"data_stored":6953140243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7524552704,"data_stored":14215929875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:23:52.007 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:23:52.007 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:23:52.008 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:23:52.008 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:23:52.009 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:24:12.010 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:24:12.508 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:24:12.509 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:24:12.569 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2244,"stamp":"2023-10-20T10:24:11.251980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27485880835,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1886423,"num_read_kb":7303097,"num_write":1845076,"num_write_kb":7378555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69853,"ondisk_log_size":69853,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19685168,"kb_used_data":14757076,"kb_used_omap":2807,"kb_used_meta":4925256,"kb_avail":263430352,"statfs":{"total":289910292480,"available":269752680448,"internally_reserved":0,"allocated":15111245824,"data_stored":28491142636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2874823,"internal_metadata":5043462713},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,7,3,8,19,17,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21569536,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5309,"num_read_kb":21156,"num_write":5259,"num_write_kb":21036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001992"},"pg_stats":[{"pgid":"2.7","version":"126'9406","reported_seq":14068,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:08.547830+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:24:08.547830+0000","last_peered":"2023-10-20T10:24:08.547830+0000","last_clean":"2023-10-20T10:24:08.547830+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:24:08.547830+0000","last_undegraded":"2023-10-20T10:24:08.547830+0000","last_fullsized":"2023-10-20T10:24:08.547830+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9406,"log_dups_size":0,"ondisk_log_size":9406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"126'18486","reported_seq":28800,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:08.565056+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:24:08.565056+0000","last_peered":"2023-10-20T10:24:08.565056+0000","last_clean":"2023-10-20T10:24:08.565056+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:24:08.565056+0000","last_undegraded":"2023-10-20T10:24:08.565056+0000","last_fullsized":"2023-10-20T10:24:08.565056+0000","mapping_epoch":13,"log_start":"96'8400","ondisk_log_start":"96'8400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":205,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:22:49.708360+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:22:49.708360+0000","last_peered":"2023-10-20T10:22:49.708360+0000","last_clean":"2023-10-20T10:22:49.708360+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:22:49.708360+0000","last_undegraded":"2023-10-20T10:22:49.708360+0000","last_fullsized":"2023-10-20T10:22:49.708360+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"126'9368","reported_seq":14804,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:08.548210+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:24:08.548210+0000","last_peered":"2023-10-20T10:24:08.548210+0000","last_clean":"2023-10-20T10:24:08.548210+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:24:08.548210+0000","last_undegraded":"2023-10-20T10:24:08.548210+0000","last_fullsized":"2023-10-20T10:24:08.548210+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9368,"log_dups_size":0,"ondisk_log_size":9368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"126'478473","reported_seq":949499,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:24:09.585353+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:24:09.585353+0000","last_peered":"2023-10-20T10:24:09.585353+0000","last_clean":"2023-10-20T10:24:09.585353+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:24:09.585353+0000","last_undegraded":"2023-10-20T10:24:09.585353+0000","last_fullsized":"2023-10-20T10:24:09.585353+0000","mapping_epoch":16,"log_start":"124'468431","ondisk_log_start":"124'468431","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":1894,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 265s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7061729280,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":474165,"num_read_kb":1867080,"num_write":473497,"num_write_kb":1893988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":299,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:22:49.194947+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:22:49.194947+0000","last_peered":"2023-10-20T10:22:49.194947+0000","last_clean":"2023-10-20T10:22:49.194947+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:22:49.194947+0000","last_undegraded":"2023-10-20T10:22:49.194947+0000","last_fullsized":"2023-10-20T10:22:49.194947+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"126'443957","reported_seq":881034,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:24:10.143574+0000","last_change":"2023-10-20T10:19:59.154102+0000","last_active":"2023-10-20T10:24:10.143574+0000","last_peered":"2023-10-20T10:24:10.143574+0000","last_clean":"2023-10-20T10:24:10.143574+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:24:10.143574+0000","last_undegraded":"2023-10-20T10:24:10.143574+0000","last_fullsized":"2023-10-20T10:24:10.143574+0000","mapping_epoch":16,"log_start":"124'433941","ondisk_log_start":"124'433941","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":2262,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 250s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6546677760,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":439966,"num_read_kb":1730308,"num_write":438934,"num_write_kb":1755736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"126'330","reported_seq":6400,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:06.806374+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:24:06.806374+0000","last_peered":"2023-10-20T10:24:06.806374+0000","last_clean":"2023-10-20T10:24:06.806374+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:24:06.806374+0000","last_undegraded":"2023-10-20T10:24:06.806374+0000","last_fullsized":"2023-10-20T10:24:06.806374+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"126'471810","reported_seq":936176,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:24:09.579483+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:24:09.579483+0000","last_peered":"2023-10-20T10:24:09.579483+0000","last_clean":"2023-10-20T10:24:09.579483+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:24:09.579483+0000","last_undegraded":"2023-10-20T10:24:09.579483+0000","last_fullsized":"2023-10-20T10:24:09.579483+0000","mapping_epoch":16,"log_start":"124'461735","ondisk_log_start":"124'461735","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":1993,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 264s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":6956232704,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":467523,"num_read_kb":1839184,"num_write":466752,"num_write_kb":1867008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"126'307","reported_seq":6250,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:06.798683+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:24:06.798683+0000","last_peered":"2023-10-20T10:24:06.798683+0000","last_clean":"2023-10-20T10:24:06.798683+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:24:06.798683+0000","last_undegraded":"2023-10-20T10:24:06.798683+0000","last_fullsized":"2023-10-20T10:24:06.798683+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"126'468353","reported_seq":931262,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:24:10.188070+0000","last_change":"2023-10-20T10:24:09.994689+0000","last_active":"2023-10-20T10:24:10.188070+0000","last_peered":"2023-10-20T10:24:10.188070+0000","last_clean":"2023-10-20T10:24:10.188070+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:24:10.188070+0000","last_undegraded":"2023-10-20T10:24:10.188070+0000","last_fullsized":"2023-10-20T10:24:10.188070+0000","mapping_epoch":16,"log_start":"124'458324","ondisk_log_start":"124'458324","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6920646675,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":466098,"num_read_kb":1833804,"num_write":464879,"num_write_kb":1859510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":204,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:22:49.708317+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:22:49.708317+0000","last_peered":"2023-10-20T10:22:49.708317+0000","last_clean":"2023-10-20T10:22:49.708317+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:22:49.708317+0000","last_undegraded":"2023-10-20T10:22:49.708317+0000","last_fullsized":"2023-10-20T10:22:49.708317+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":298,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:23:43.091075+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:23:43.091075+0000","last_peered":"2023-10-20T10:23:43.091075+0000","last_clean":"2023-10-20T10:23:43.091075+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:23:43.091075+0000","last_undegraded":"2023-10-20T10:23:43.091075+0000","last_fullsized":"2023-10-20T10:23:43.091075+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27485286419,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1847752,"num_read_kb":7270376,"num_write":1844062,"num_write_kb":7376242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15106125824,"data_stored":28488880166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40162,"ondisk_log_size":40162,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":29592,"ondisk_log_size":29592,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739218,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9089280,"kb_used_data":7377016,"kb_used_omap":1432,"kb_used_meta":1710823,"kb_avail":85282560,"statfs":{"total":96636764160,"available":87329341440,"internally_reserved":0,"allocated":7554064384,"data_stored":14244595505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466972,"internal_metadata":1751883172},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.38,"15min":0.44600000000000001},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.875,"5min":3.9870000000000001,"15min":8.2449999999999992},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.41299999999999998,"15min":0.47899999999999998},"min":{"1min":0.21299999999999999,"5min":0.20499999999999999,"15min":0.18099999999999999},"max":{"1min":0.73299999999999998,"5min":4.9749999999999996,"15min":6.8239999999999998},"last":0.33500000000000002}]},{"osd":1,"last update":"Fri Oct 20 10:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.48099999999999998,"15min":0.69299999999999995},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":1.1990000000000001,"5min":4.8659999999999997,"15min":35.670000000000002},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.44400000000000001,"15min":0.58499999999999996},"min":{"1min":0.245,"5min":0.217,"15min":0.214},"max":{"1min":1.232,"5min":4.899,"15min":34.948999999999998},"last":0.378}]}]},{"osd":1,"up_from":8,"seq":34359739218,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5310296,"kb_used_data":3617232,"kb_used_omap":676,"kb_used_meta":1692379,"kb_avail":89061544,"statfs":{"total":96636764160,"available":91199021056,"internally_reserved":0,"allocated":3704045568,"data_stored":6968238757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1732996683},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,0,3,5,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.51000000000000001,"15min":0.45800000000000002},"min":{"1min":0.188,"5min":0.17899999999999999,"15min":0.157},"max":{"1min":4.7510000000000003,"5min":4.7510000000000003,"15min":4.7510000000000003},"last":0.19600000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.436,"15min":0.44500000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":5.4640000000000004,"5min":5.4640000000000004,"15min":5.8479999999999999},"last":0.252}]},{"osd":2,"last update":"Fri Oct 20 10:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.41199999999999998,"15min":0.40899999999999997},"min":{"1min":0.22500000000000001,"5min":0.19900000000000001,"15min":0.18099999999999999},"max":{"1min":4.7859999999999996,"5min":4.7859999999999996,"15min":4.8920000000000003},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.45800000000000002,"15min":0.45400000000000001},"min":{"1min":0.26400000000000001,"5min":0.24099999999999999,"15min":0.23799999999999999},"max":{"1min":4.8109999999999999,"5min":4.8109999999999999,"15min":4.9279999999999999},"last":0.317}]}]},{"osd":0,"up_from":8,"seq":34359739219,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5285592,"kb_used_data":3762828,"kb_used_omap":698,"kb_used_meta":1522053,"kb_avail":89086248,"statfs":{"total":96636764160,"available":91224317952,"internally_reserved":0,"allocated":3853135872,"data_stored":7278308374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":1558582858},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,5,14,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:23:46 2023","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":0.64900000000000002,"15min":0.82499999999999996},"min":{"1min":0.17899999999999999,"5min":0.17799999999999999,"15min":0.16700000000000001},"max":{"1min":8.0869999999999997,"5min":8.0869999999999997,"15min":81.197000000000003},"last":0.214},{"interface":"front","average":{"1min":1.052,"5min":0.59199999999999997,"15min":0.72399999999999998},"min":{"1min":0.21299999999999999,"5min":0.20000000000000001,"15min":0.184},"max":{"1min":6.875,"5min":7.2690000000000001,"15min":80.334999999999994},"last":0.27200000000000002}]},{"osd":2,"last update":"Fri Oct 20 10:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.40699999999999997,"15min":0.51900000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.53100000000000003,"5min":3.653,"15min":11.176},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.40200000000000002,"15min":0.58899999999999997},"min":{"1min":0.20599999999999999,"5min":0.20000000000000001,"15min":0.187},"max":{"1min":1.093,"5min":3.4209999999999998,"15min":11.093},"last":0.47299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3851264000,"data_stored":7277367296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3702022144,"data_stored":6967296019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7552839680,"data_stored":14244216851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:24:12.571 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:24:12.571 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:24:12.572 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:24:12.572 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:24:12.573 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-20T10:24:13.159 INFO:teuthology.orchestra.run.smithi150.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-20T10:24:13.192 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-20T10:24:13.835 INFO:teuthology.orchestra.run.smithi150.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-20T10:24:13.883 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-20T10:24:14.381 INFO:teuthology.orchestra.run.smithi150.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-20T10:24:14.418 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-20T10:24:14.881 INFO:teuthology.orchestra.run.smithi150.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-20T10:24:14.921 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:24:34.922 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:24:35.624 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:24:35.625 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:24:35.670 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2256,"stamp":"2023-10-20T10:24:35.257492+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27518853635,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1894739,"num_read_kb":7336177,"num_write":1853125,"num_write_kb":7410751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69822,"ondisk_log_size":69822,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20004340,"kb_used_data":14824860,"kb_used_omap":2807,"kb_used_meta":5176648,"kb_avail":263111180,"statfs":{"total":289910292480,"available":269425848320,"internally_reserved":0,"allocated":15180656640,"data_stored":28560553452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2874823,"internal_metadata":5300888121},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,4,11,15,13],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18665472,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4694,"num_read_kb":18664,"num_write":4554,"num_write_kb":18216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003316"},"pg_stats":[{"pgid":"2.7","version":"126'9427","reported_seq":14101,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:27.278011+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:24:27.278011+0000","last_peered":"2023-10-20T10:24:27.278011+0000","last_clean":"2023-10-20T10:24:27.278011+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:24:27.278011+0000","last_undegraded":"2023-10-20T10:24:27.278011+0000","last_fullsized":"2023-10-20T10:24:27.278011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9427,"log_dups_size":0,"ondisk_log_size":9427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"126'18528","reported_seq":28866,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:27.278801+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:24:27.278801+0000","last_peered":"2023-10-20T10:24:27.278801+0000","last_clean":"2023-10-20T10:24:27.278801+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:24:27.278801+0000","last_undegraded":"2023-10-20T10:24:27.278801+0000","last_fullsized":"2023-10-20T10:24:27.278801+0000","mapping_epoch":13,"log_start":"98'8500","ondisk_log_start":"98'8500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":205,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:22:49.708360+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:22:49.708360+0000","last_peered":"2023-10-20T10:22:49.708360+0000","last_clean":"2023-10-20T10:22:49.708360+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:22:49.708360+0000","last_undegraded":"2023-10-20T10:22:49.708360+0000","last_fullsized":"2023-10-20T10:22:49.708360+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"126'9389","reported_seq":14837,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:27.278152+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:24:27.278152+0000","last_peered":"2023-10-20T10:24:27.278152+0000","last_clean":"2023-10-20T10:24:27.278152+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:24:27.278152+0000","last_undegraded":"2023-10-20T10:24:27.278152+0000","last_fullsized":"2023-10-20T10:24:27.278152+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9389,"log_dups_size":0,"ondisk_log_size":9389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"126'480846","reported_seq":954243,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:24:34.795487+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:24:34.795487+0000","last_peered":"2023-10-20T10:24:34.795487+0000","last_clean":"2023-10-20T10:24:34.795487+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:24:34.795487+0000","last_undegraded":"2023-10-20T10:24:34.795487+0000","last_fullsized":"2023-10-20T10:24:34.795487+0000","mapping_epoch":16,"log_start":"125'470831","ondisk_log_start":"125'470831","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":2059,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 290s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7070961664,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":476520,"num_read_kb":1876444,"num_write":475745,"num_write_kb":1902980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":299,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:22:49.194947+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:22:49.194947+0000","last_peered":"2023-10-20T10:22:49.194947+0000","last_clean":"2023-10-20T10:22:49.194947+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:22:49.194947+0000","last_undegraded":"2023-10-20T10:22:49.194947+0000","last_fullsized":"2023-10-20T10:22:49.194947+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"126'445770","reported_seq":884623,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:24:30.264167+0000","last_change":"2023-10-20T10:19:59.154102+0000","last_active":"2023-10-20T10:24:30.264167+0000","last_peered":"2023-10-20T10:24:30.264167+0000","last_clean":"2023-10-20T10:24:30.264167+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:24:30.264167+0000","last_undegraded":"2023-10-20T10:24:30.264167+0000","last_fullsized":"2023-10-20T10:24:30.264167+0000","mapping_epoch":16,"log_start":"125'435741","ondisk_log_start":"125'435741","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":2417,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 271s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6553632768,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":441730,"num_read_kb":1737328,"num_write":440631,"num_write_kb":1762524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"126'330","reported_seq":6420,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:32.402378+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:24:32.402378+0000","last_peered":"2023-10-20T10:24:32.402378+0000","last_clean":"2023-10-20T10:24:32.402378+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:24:32.402378+0000","last_undegraded":"2023-10-20T10:24:32.402378+0000","last_fullsized":"2023-10-20T10:24:32.402378+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"126'474208","reported_seq":940896,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:24:34.787388+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:24:34.787388+0000","last_peered":"2023-10-20T10:24:34.787388+0000","last_clean":"2023-10-20T10:24:34.787388+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:24:34.787388+0000","last_undegraded":"2023-10-20T10:24:34.787388+0000","last_fullsized":"2023-10-20T10:24:34.787388+0000","mapping_epoch":16,"log_start":"125'464135","ondisk_log_start":"125'464135","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":2116,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 289s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":6965645312,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":469827,"num_read_kb":1848352,"num_write":469055,"num_write_kb":1876220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"126'307","reported_seq":6270,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:32.398277+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:24:32.398277+0000","last_peered":"2023-10-20T10:24:32.398277+0000","last_clean":"2023-10-20T10:24:32.398277+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:24:32.398277+0000","last_undegraded":"2023-10-20T10:24:32.398277+0000","last_fullsized":"2023-10-20T10:24:32.398277+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"126'470154","reported_seq":934967,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:30.271223+0000","last_change":"2023-10-20T10:24:27.124514+0000","last_active":"2023-10-20T10:24:30.271223+0000","last_peered":"2023-10-20T10:24:30.271223+0000","last_clean":"2023-10-20T10:24:30.271223+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:24:30.271223+0000","last_undegraded":"2023-10-20T10:24:30.271223+0000","last_fullsized":"2023-10-20T10:24:30.271223+0000","mapping_epoch":16,"log_start":"125'460124","ondisk_log_start":"125'460124","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6928019475,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":467991,"num_read_kb":1841332,"num_write":466680,"num_write_kb":1866714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":204,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:22:49.708317+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:22:49.708317+0000","last_peered":"2023-10-20T10:22:49.708317+0000","last_clean":"2023-10-20T10:22:49.708317+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:22:49.708317+0000","last_undegraded":"2023-10-20T10:22:49.708317+0000","last_fullsized":"2023-10-20T10:22:49.708317+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":298,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:23:43.091075+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:23:43.091075+0000","last_peered":"2023-10-20T10:23:43.091075+0000","last_clean":"2023-10-20T10:23:43.091075+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:23:43.091075+0000","last_undegraded":"2023-10-20T10:23:43.091075+0000","last_fullsized":"2023-10-20T10:23:43.091075+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27518259219,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1856068,"num_read_kb":7303456,"num_write":1852111,"num_write_kb":7408438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15175569408,"data_stored":28558323750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40147,"ondisk_log_size":40147,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":29576,"ondisk_log_size":29576,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739223,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9279076,"kb_used_data":7412444,"kb_used_omap":1432,"kb_used_meta":1865191,"kb_avail":85092764,"statfs":{"total":96636764160,"available":87134990336,"internally_reserved":0,"allocated":7590342656,"data_stored":14280873777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466972,"internal_metadata":1909956004},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,4,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.38,"15min":0.44600000000000001},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.875,"5min":3.9870000000000001,"15min":8.2449999999999992},"last":0.498},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.41299999999999998,"15min":0.47899999999999998},"min":{"1min":0.21299999999999999,"5min":0.20499999999999999,"15min":0.18099999999999999},"max":{"1min":0.73299999999999998,"5min":4.9749999999999996,"15min":6.8239999999999998},"last":0.46500000000000002}]},{"osd":1,"last update":"Fri Oct 20 10:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.48099999999999998,"15min":0.69299999999999995},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":1.1990000000000001,"5min":4.8659999999999997,"15min":35.670000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.44400000000000001,"15min":0.58499999999999996},"min":{"1min":0.245,"5min":0.217,"15min":0.214},"max":{"1min":1.232,"5min":4.899,"15min":34.948999999999998},"last":0.437}]}]},{"osd":1,"up_from":8,"seq":34359739222,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5360484,"kb_used_data":3631324,"kb_used_omap":676,"kb_used_meta":1728475,"kb_avail":89011356,"statfs":{"total":96636764160,"available":91147628544,"internally_reserved":0,"allocated":3718475776,"data_stored":6982668965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1769958987},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.51000000000000001,"15min":0.45800000000000002},"min":{"1min":0.188,"5min":0.17899999999999999,"15min":0.157},"max":{"1min":4.7510000000000003,"5min":4.7510000000000003,"15min":4.7510000000000003},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.436,"15min":0.44500000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":5.4640000000000004,"5min":5.4640000000000004,"15min":5.8479999999999999},"last":0.34000000000000002}]},{"osd":2,"last update":"Fri Oct 20 10:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.41199999999999998,"15min":0.40899999999999997},"min":{"1min":0.22500000000000001,"5min":0.19900000000000001,"15min":0.18099999999999999},"max":{"1min":4.7859999999999996,"5min":4.7859999999999996,"15min":4.8920000000000003},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.45800000000000002,"15min":0.45400000000000001},"min":{"1min":0.26400000000000001,"5min":0.24099999999999999,"15min":0.23799999999999999},"max":{"1min":4.8109999999999999,"5min":4.8109999999999999,"15min":4.9279999999999999},"last":0.41999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739225,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5364780,"kb_used_data":3781092,"kb_used_omap":698,"kb_used_meta":1582981,"kb_avail":89007060,"statfs":{"total":96636764160,"available":91143229440,"internally_reserved":0,"allocated":3871838208,"data_stored":7297010710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":1620973130},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,7,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:23:46 2023","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":0.64900000000000002,"15min":0.82499999999999996},"min":{"1min":0.17899999999999999,"5min":0.17799999999999999,"15min":0.16700000000000001},"max":{"1min":8.0869999999999997,"5min":8.0869999999999997,"15min":81.197000000000003},"last":0.29699999999999999},{"interface":"front","average":{"1min":1.052,"5min":0.59199999999999997,"15min":0.72399999999999998},"min":{"1min":0.21299999999999999,"5min":0.20000000000000001,"15min":0.184},"max":{"1min":6.875,"5min":7.2690000000000001,"15min":80.334999999999994},"last":0.26700000000000002}]},{"osd":2,"last update":"Fri Oct 20 10:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.40699999999999997,"15min":0.51900000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.53100000000000003,"5min":3.653,"15min":11.176},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.40200000000000002,"15min":0.58899999999999997},"min":{"1min":0.20599999999999999,"5min":0.20000000000000001,"15min":0.187},"max":{"1min":1.093,"5min":3.4209999999999998,"15min":11.093},"last":0.34200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3869900800,"data_stored":7296004096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3716341760,"data_stored":6981615635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7589326848,"data_stored":14280704019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:24:35.672 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:24:35.672 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:24:35.673 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:24:35.673 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:24:35.674 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:24:37.383 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:24:37.380+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:24:37.394 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:24:37.388+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:24:55.675 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:24:56.173 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:24:56.174 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:24:56.212 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2266,"stamp":"2023-10-20T10:24:55.260845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27547357699,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1901885,"num_read_kb":7364637,"num_write":1860087,"num_write_kb":7438599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69896,"ondisk_log_size":69896,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20201676,"kb_used_data":14879920,"kb_used_omap":2807,"kb_used_meta":5318920,"kb_avail":262913844,"statfs":{"total":289910292480,"available":269223776256,"internally_reserved":0,"allocated":15237038080,"data_stored":28616934892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2874823,"internal_metadata":5446574649},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,3,6,4,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17997824,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4542,"num_read_kb":18092,"num_write":4394,"num_write_kb":17576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001965"},"pg_stats":[{"pgid":"2.7","version":"126'9470","reported_seq":14156,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:48.968965+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:24:48.968965+0000","last_peered":"2023-10-20T10:24:48.968965+0000","last_clean":"2023-10-20T10:24:48.968965+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:24:48.968965+0000","last_undegraded":"2023-10-20T10:24:48.968965+0000","last_fullsized":"2023-10-20T10:24:48.968965+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9470,"log_dups_size":0,"ondisk_log_size":9470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"126'18613","reported_seq":28975,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:48.995883+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:24:48.995883+0000","last_peered":"2023-10-20T10:24:48.995883+0000","last_clean":"2023-10-20T10:24:48.995883+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:24:48.995883+0000","last_undegraded":"2023-10-20T10:24:48.995883+0000","last_fullsized":"2023-10-20T10:24:48.995883+0000","mapping_epoch":13,"log_start":"98'8600","ondisk_log_start":"98'8600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":205,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:22:49.708360+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:22:49.708360+0000","last_peered":"2023-10-20T10:22:49.708360+0000","last_clean":"2023-10-20T10:22:49.708360+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:22:49.708360+0000","last_undegraded":"2023-10-20T10:22:49.708360+0000","last_fullsized":"2023-10-20T10:22:49.708360+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"126'9432","reported_seq":14892,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:48.959239+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:24:48.959239+0000","last_peered":"2023-10-20T10:24:48.959239+0000","last_clean":"2023-10-20T10:24:48.959239+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:24:48.959239+0000","last_undegraded":"2023-10-20T10:24:48.959239+0000","last_fullsized":"2023-10-20T10:24:48.959239+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9432,"log_dups_size":0,"ondisk_log_size":9432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"126'482772","reported_seq":958023,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:24:54.850941+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:24:54.850941+0000","last_peered":"2023-10-20T10:24:54.850941+0000","last_clean":"2023-10-20T10:24:54.850941+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:24:54.850941+0000","last_undegraded":"2023-10-20T10:24:54.850941+0000","last_fullsized":"2023-10-20T10:24:54.850941+0000","mapping_epoch":16,"log_start":"125'472731","ondisk_log_start":"125'472731","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":2196,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 310s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7078440960,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":478360,"num_read_kb":1883772,"num_write":477570,"num_write_kb":1910280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":300,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:23:43.099383+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:23:43.099383+0000","last_peered":"2023-10-20T10:23:43.099383+0000","last_clean":"2023-10-20T10:23:43.099383+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:23:43.099383+0000","last_undegraded":"2023-10-20T10:23:43.099383+0000","last_fullsized":"2023-10-20T10:23:43.099383+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"126'447644","reported_seq":888203,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:24:50.371480+0000","last_change":"2023-10-20T10:19:59.154102+0000","last_active":"2023-10-20T10:24:50.371480+0000","last_peered":"2023-10-20T10:24:50.371480+0000","last_clean":"2023-10-20T10:24:50.371480+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:24:50.371480+0000","last_undegraded":"2023-10-20T10:24:50.371480+0000","last_fullsized":"2023-10-20T10:24:50.371480+0000","mapping_epoch":16,"log_start":"125'437641","ondisk_log_start":"125'437641","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":2642,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 291s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6560600064,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":443420,"num_read_kb":1744060,"num_write":442332,"num_write_kb":1769328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"126'330","reported_seq":6437,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:52.879267+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:24:52.879267+0000","last_peered":"2023-10-20T10:24:52.879267+0000","last_clean":"2023-10-20T10:24:52.879267+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:24:52.879267+0000","last_undegraded":"2023-10-20T10:24:52.879267+0000","last_fullsized":"2023-10-20T10:24:52.879267+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"126'476096","reported_seq":944599,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:24:54.855250+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:24:54.855250+0000","last_peered":"2023-10-20T10:24:54.855250+0000","last_clean":"2023-10-20T10:24:54.855250+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:24:54.855250+0000","last_undegraded":"2023-10-20T10:24:54.855250+0000","last_fullsized":"2023-10-20T10:24:54.855250+0000","mapping_epoch":16,"log_start":"125'466035","ondisk_log_start":"125'466035","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":2329,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 309s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":6972686336,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":471633,"num_read_kb":1855540,"num_write":470776,"num_write_kb":1883104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"126'307","reported_seq":6286,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:52.895598+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:24:52.895598+0000","last_peered":"2023-10-20T10:24:52.895598+0000","last_clean":"2023-10-20T10:24:52.895598+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:24:52.895598+0000","last_undegraded":"2023-10-20T10:24:52.895598+0000","last_fullsized":"2023-10-20T10:24:52.895598+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"126'471869","reported_seq":938497,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:50.355670+0000","last_change":"2023-10-20T10:24:47.040583+0000","last_active":"2023-10-20T10:24:50.355670+0000","last_peered":"2023-10-20T10:24:50.355670+0000","last_clean":"2023-10-20T10:24:50.355670+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:24:50.355670+0000","last_undegraded":"2023-10-20T10:24:50.355670+0000","last_fullsized":"2023-10-20T10:24:50.355670+0000","mapping_epoch":16,"log_start":"125'461824","ondisk_log_start":"125'461824","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6935035923,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":469801,"num_read_kb":1848544,"num_write":468395,"num_write_kb":1873574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":204,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-20T10:22:49.708317+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:22:49.708317+0000","last_peered":"2023-10-20T10:22:49.708317+0000","last_clean":"2023-10-20T10:22:49.708317+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:22:49.708317+0000","last_undegraded":"2023-10-20T10:22:49.708317+0000","last_fullsized":"2023-10-20T10:22:49.708317+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":298,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:23:43.091075+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:23:43.091075+0000","last_peered":"2023-10-20T10:23:43.091075+0000","last_clean":"2023-10-20T10:23:43.091075+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:23:43.091075+0000","last_undegraded":"2023-10-20T10:23:43.091075+0000","last_fullsized":"2023-10-20T10:23:43.091075+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27546763283,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1863214,"num_read_kb":7331916,"num_write":1859073,"num_write_kb":7436286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15232610304,"data_stored":28615364646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40150,"ondisk_log_size":40150,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":29647,"ondisk_log_size":29647,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739227,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9375108,"kb_used_data":7440444,"kb_used_omap":1432,"kb_used_meta":1933223,"kb_avail":84996732,"statfs":{"total":96636764160,"available":87036653568,"internally_reserved":0,"allocated":7619014656,"data_stored":14309545777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466972,"internal_metadata":1979620772},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.38,"15min":0.44600000000000001},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.875,"5min":3.9870000000000001,"15min":8.2449999999999992},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.41299999999999998,"15min":0.47899999999999998},"min":{"1min":0.21299999999999999,"5min":0.20499999999999999,"15min":0.18099999999999999},"max":{"1min":0.73299999999999998,"5min":4.9749999999999996,"15min":6.8239999999999998},"last":0.28899999999999998}]},{"osd":1,"last update":"Fri Oct 20 10:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.48099999999999998,"15min":0.69299999999999995},"min":{"1min":0.22900000000000001,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":1.1990000000000001,"5min":4.8659999999999997,"15min":35.670000000000002},"last":0.25},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.44400000000000001,"15min":0.58499999999999996},"min":{"1min":0.245,"5min":0.217,"15min":0.214},"max":{"1min":1.232,"5min":4.899,"15min":34.948999999999998},"last":0.32000000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739226,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5412192,"kb_used_data":3644824,"kb_used_omap":676,"kb_used_meta":1766683,"kb_avail":88959648,"statfs":{"total":96636764160,"available":91094679552,"internally_reserved":0,"allocated":3732299776,"data_stored":6996492965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1809083979},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.51000000000000001,"15min":0.45800000000000002},"min":{"1min":0.188,"5min":0.17899999999999999,"15min":0.157},"max":{"1min":4.7510000000000003,"5min":4.7510000000000003,"15min":4.7510000000000003},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.436,"15min":0.44500000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":5.4640000000000004,"5min":5.4640000000000004,"15min":5.8479999999999999},"last":0.42599999999999999}]},{"osd":2,"last update":"Fri Oct 20 10:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.41199999999999998,"15min":0.40899999999999997},"min":{"1min":0.22500000000000001,"5min":0.19900000000000001,"15min":0.18099999999999999},"max":{"1min":4.7859999999999996,"5min":4.7859999999999996,"15min":4.8920000000000003},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.45800000000000002,"15min":0.45400000000000001},"min":{"1min":0.26400000000000001,"5min":0.24099999999999999,"15min":0.23799999999999999},"max":{"1min":4.8109999999999999,"5min":4.8109999999999999,"15min":4.9279999999999999},"last":0.31900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739229,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5414376,"kb_used_data":3794652,"kb_used_omap":698,"kb_used_meta":1619013,"kb_avail":88957464,"statfs":{"total":96636764160,"available":91092443136,"internally_reserved":0,"allocated":3885723648,"data_stored":7310896150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":1657869898},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:24:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.70399999999999996,"15min":0.81000000000000005},"min":{"1min":0.182,"5min":0.17799999999999999,"15min":0.16700000000000001},"max":{"1min":2.3290000000000002,"5min":8.0869999999999997,"15min":81.197000000000003},"last":0.188},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.54700000000000004,"15min":0.72699999999999998},"min":{"1min":0.22,"5min":0.20000000000000001,"15min":0.184},"max":{"1min":2.363,"5min":6.875,"15min":80.334999999999994},"last":0.222}]},{"osd":2,"last update":"Fri Oct 20 10:24:47 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.39700000000000002,"15min":0.504},"min":{"1min":0.216,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":2.3969999999999998,"5min":3.653,"15min":11.176},"last":0.311},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.379,"15min":0.54100000000000004},"min":{"1min":0.21199999999999999,"5min":0.20000000000000001,"15min":0.187},"max":{"1min":1.0289999999999999,"5min":1.5049999999999999,"15min":11.093},"last":0.33600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3884429312,"data_stored":7310532608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3730333696,"data_stored":6995607571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7617847296,"data_stored":14309224467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:24:56.214 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:24:56.215 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:24:56.215 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:24:56.215 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:24:56.216 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:25:16.217 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:25:16.759 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:25:16.761 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:25:16.791 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2277,"stamp":"2023-10-20T10:25:15.264305+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27576541699,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1909004,"num_read_kb":7392921,"num_write":1867207,"num_write_kb":7467079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70102,"ondisk_log_size":70102,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20126272,"kb_used_data":14936808,"kb_used_omap":2807,"kb_used_meta":5186632,"kb_avail":262989248,"statfs":{"total":289910292480,"available":269300989952,"internally_reserved":0,"allocated":15295291392,"data_stored":28675165035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2874823,"internal_metadata":5311111737},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,3,8,9,16,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18440192,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4499,"num_read_kb":17880,"num_write":4495,"num_write_kb":17980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002021"},"pg_stats":[{"pgid":"2.7","version":"127'9506","reported_seq":14205,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:25:08.708155+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:25:08.708155+0000","last_peered":"2023-10-20T10:25:08.708155+0000","last_clean":"2023-10-20T10:25:08.708155+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:25:08.708155+0000","last_undegraded":"2023-10-20T10:25:08.708155+0000","last_fullsized":"2023-10-20T10:25:08.708155+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9506,"log_dups_size":0,"ondisk_log_size":9506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"127'18685","reported_seq":29072,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:25:08.728603+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:25:08.728603+0000","last_peered":"2023-10-20T10:25:08.728603+0000","last_clean":"2023-10-20T10:25:08.728603+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:25:08.728603+0000","last_undegraded":"2023-10-20T10:25:08.728603+0000","last_fullsized":"2023-10-20T10:25:08.728603+0000","mapping_epoch":13,"log_start":"98'8600","ondisk_log_start":"98'8600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":207,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:55.428541+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:24:55.428541+0000","last_peered":"2023-10-20T10:24:55.428541+0000","last_clean":"2023-10-20T10:24:55.428541+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:24:55.428541+0000","last_undegraded":"2023-10-20T10:24:55.428541+0000","last_fullsized":"2023-10-20T10:24:55.428541+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"127'9468","reported_seq":14942,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:25:08.698387+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:25:08.698387+0000","last_peered":"2023-10-20T10:25:08.698387+0000","last_clean":"2023-10-20T10:25:08.698387+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:25:08.698387+0000","last_undegraded":"2023-10-20T10:25:08.698387+0000","last_fullsized":"2023-10-20T10:25:08.698387+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9468,"log_dups_size":0,"ondisk_log_size":9468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"127'484665","reported_seq":961835,"reported_epoch":127,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:25:14.953746+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:25:14.953746+0000","last_peered":"2023-10-20T10:25:14.953746+0000","last_clean":"2023-10-20T10:25:14.953746+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:25:14.953746+0000","last_undegraded":"2023-10-20T10:25:14.953746+0000","last_fullsized":"2023-10-20T10:25:14.953746+0000","mapping_epoch":16,"log_start":"125'474631","ondisk_log_start":"125'474631","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":2321,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 331s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7085830144,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":480266,"num_read_kb":1891364,"num_write":479367,"num_write_kb":1917468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":301,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:54.855620+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:24:54.855620+0000","last_peered":"2023-10-20T10:24:54.855620+0000","last_clean":"2023-10-20T10:24:54.855620+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:24:54.855620+0000","last_undegraded":"2023-10-20T10:24:54.855620+0000","last_fullsized":"2023-10-20T10:24:54.855620+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"127'449551","reported_seq":891851,"reported_epoch":127,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:25:10.537449+0000","last_change":"2023-10-20T10:19:59.154102+0000","last_active":"2023-10-20T10:25:10.537449+0000","last_peered":"2023-10-20T10:25:10.537449+0000","last_clean":"2023-10-20T10:25:10.537449+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:25:10.537449+0000","last_undegraded":"2023-10-20T10:25:10.537449+0000","last_fullsized":"2023-10-20T10:25:10.537449+0000","mapping_epoch":16,"log_start":"125'439541","ondisk_log_start":"125'439541","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":2834,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 311s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6567817216,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":445146,"num_read_kb":1750916,"num_write":444092,"num_write_kb":1776368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"126'330","reported_seq":6454,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:25:13.366323+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:25:13.366323+0000","last_peered":"2023-10-20T10:25:13.366323+0000","last_clean":"2023-10-20T10:25:13.366323+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:25:13.366323+0000","last_undegraded":"2023-10-20T10:25:13.366323+0000","last_fullsized":"2023-10-20T10:25:13.366323+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"127'478003","reported_seq":948254,"reported_epoch":127,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:25:14.922722+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:25:14.922722+0000","last_peered":"2023-10-20T10:25:14.922722+0000","last_clean":"2023-10-20T10:25:14.922722+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:25:14.922722+0000","last_undegraded":"2023-10-20T10:25:14.922722+0000","last_fullsized":"2023-10-20T10:25:14.922722+0000","mapping_epoch":16,"log_start":"125'467935","ondisk_log_start":"125'467935","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":2456,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 329s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":6980087808,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":473365,"num_read_kb":1862408,"num_write":472584,"num_write_kb":1890336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"126'307","reported_seq":6303,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:25:13.446371+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:25:13.446371+0000","last_peered":"2023-10-20T10:25:13.446371+0000","last_clean":"2023-10-20T10:25:13.446371+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:25:13.446371+0000","last_undegraded":"2023-10-20T10:25:13.446371+0000","last_fullsized":"2023-10-20T10:25:13.446371+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"127'473624","reported_seq":942014,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:25:10.544512+0000","last_change":"2023-10-20T10:25:08.942116+0000","last_active":"2023-10-20T10:25:10.544512+0000","last_peered":"2023-10-20T10:25:10.544512+0000","last_clean":"2023-10-20T10:25:10.544512+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:25:10.544512+0000","last_undegraded":"2023-10-20T10:25:10.544512+0000","last_fullsized":"2023-10-20T10:25:10.544512+0000","mapping_epoch":16,"log_start":"125'463524","ondisk_log_start":"125'463524","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6942212115,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":471556,"num_read_kb":1855512,"num_write":470150,"num_write_kb":1880594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":206,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:55.428505+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:24:55.428505+0000","last_peered":"2023-10-20T10:24:55.428505+0000","last_clean":"2023-10-20T10:24:55.428505+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:24:55.428505+0000","last_undegraded":"2023-10-20T10:24:55.428505+0000","last_fullsized":"2023-10-20T10:24:55.428505+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":299,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:00.113495+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:24:00.113495+0000","last_peered":"2023-10-20T10:24:00.113495+0000","last_clean":"2023-10-20T10:24:00.113495+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:24:00.113495+0000","last_undegraded":"2023-10-20T10:24:00.113495+0000","last_fullsized":"2023-10-20T10:24:00.113495+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27575947283,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1870333,"num_read_kb":7360200,"num_write":1866193,"num_write_kb":7464766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15290871808,"data_stored":28673626150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40212,"ondisk_log_size":40212,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":29791,"ondisk_log_size":29791,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739231,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9174220,"kb_used_data":7468868,"kb_used_omap":1432,"kb_used_meta":1703911,"kb_avail":85197620,"statfs":{"total":96636764160,"available":87242362880,"internally_reserved":0,"allocated":7648120832,"data_stored":14338644230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466972,"internal_metadata":1744805284},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:25:06 2023","interfaces":[{"interface":"back","average":{"1min":0.28399999999999997,"5min":0.33500000000000002,"15min":0.443},"min":{"1min":0.16800000000000001,"5min":0.16700000000000001,"15min":0.16200000000000001},"max":{"1min":0.498,"5min":1.806,"15min":8.2449999999999992},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.36499999999999999,"15min":0.47599999999999998},"min":{"1min":0.24299999999999999,"5min":0.20499999999999999,"15min":0.18099999999999999},"max":{"1min":0.56100000000000005,"5min":2.6419999999999999,"15min":6.8239999999999998},"last":0.29699999999999999}]},{"osd":1,"last update":"Fri Oct 20 10:25:06 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.443,"15min":0.67700000000000005},"min":{"1min":0.19800000000000001,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":0.55600000000000005,"5min":3.5150000000000001,"15min":35.670000000000002},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.40000000000000002,"15min":0.56399999999999995},"min":{"1min":0.22900000000000001,"5min":0.22500000000000001,"15min":0.214},"max":{"1min":0.50800000000000001,"5min":2.7090000000000001,"15min":34.948999999999998},"last":0.32900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739230,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5462156,"kb_used_data":3658692,"kb_used_omap":676,"kb_used_meta":1802779,"kb_avail":88909684,"statfs":{"total":96636764160,"available":91043516416,"internally_reserved":0,"allocated":3746500608,"data_stored":7010686074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1846046283},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:24:53 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.45200000000000001,"15min":0.46200000000000002},"min":{"1min":0.19600000000000001,"5min":0.17899999999999999,"15min":0.157},"max":{"1min":1.212,"5min":4.7510000000000003,"15min":4.7510000000000003},"last":0.308},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.45300000000000001,"15min":0.46100000000000002},"min":{"1min":0.19800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":5.2560000000000002,"5min":5.4640000000000004,"15min":5.8479999999999999},"last":0.25800000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:24:53 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.40999999999999998,"15min":0.39900000000000002},"min":{"1min":0.222,"5min":0.19900000000000001,"15min":0.18099999999999999},"max":{"1min":0.58099999999999996,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.45800000000000002,"15min":0.44500000000000001},"min":{"1min":0.27400000000000002,"5min":0.252,"15min":0.23799999999999999},"max":{"1min":0.61199999999999999,"5min":4.8109999999999999,"15min":4.8109999999999999},"last":0.38300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739233,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5489896,"kb_used_data":3809248,"kb_used_omap":698,"kb_used_meta":1679941,"kb_avail":88881944,"statfs":{"total":96636764160,"available":91015110656,"internally_reserved":0,"allocated":3900669952,"data_stored":7325834731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":1720260170},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,5,3,11,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:24:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.70399999999999996,"15min":0.81000000000000005},"min":{"1min":0.182,"5min":0.17799999999999999,"15min":0.16700000000000001},"max":{"1min":2.3290000000000002,"5min":8.0869999999999997,"15min":81.197000000000003},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.54700000000000004,"15min":0.72699999999999998},"min":{"1min":0.22,"5min":0.20000000000000001,"15min":0.184},"max":{"1min":2.363,"5min":6.875,"15min":80.334999999999994},"last":0.317}]},{"osd":2,"last update":"Fri Oct 20 10:24:47 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.39700000000000002,"15min":0.504},"min":{"1min":0.216,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":2.3969999999999998,"5min":3.653,"15min":11.176},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.379,"15min":0.54100000000000004},"min":{"1min":0.21199999999999999,"5min":0.20000000000000001,"15min":0.187},"max":{"1min":1.0289999999999999,"5min":1.5049999999999999,"15min":11.093},"last":0.247}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3899224064,"data_stored":7325327360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3744735232,"data_stored":7010009107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7646912512,"data_stored":14338289683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:25:16.793 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:25:16.793 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:25:16.794 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:25:16.794 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:25:16.795 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:25:18.967 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:25:18.967+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:25:18.979 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:25:18.979+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:25:18.982 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:25:18.983+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:25:36.796 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:25:37.367 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:25:37.368 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:25:37.400 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2288,"stamp":"2023-10-20T10:25:37.268055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27609850371,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1917051,"num_read_kb":7424885,"num_write":1875335,"num_write_kb":7499591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70125,"ondisk_log_size":70125,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20349244,"kb_used_data":14998564,"kb_used_omap":2807,"kb_used_meta":5347848,"kb_avail":262766276,"statfs":{"total":289910292480,"available":269072666624,"internally_reserved":0,"allocated":15358529536,"data_stored":28738403179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2874823,"internal_metadata":5476196921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,6,2,11,15,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18792448,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4528,"num_read_kb":17980,"num_write":4590,"num_write_kb":18360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002002"},"pg_stats":[{"pgid":"2.7","version":"127'9532","reported_seq":14247,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:25:33.842200+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:25:33.842200+0000","last_peered":"2023-10-20T10:25:33.842200+0000","last_clean":"2023-10-20T10:25:33.842200+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:25:33.842200+0000","last_undegraded":"2023-10-20T10:25:33.842200+0000","last_fullsized":"2023-10-20T10:25:33.842200+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9532,"log_dups_size":0,"ondisk_log_size":9532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"127'18734","reported_seq":29152,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:25:33.843174+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:25:33.843174+0000","last_peered":"2023-10-20T10:25:33.843174+0000","last_clean":"2023-10-20T10:25:33.843174+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:25:33.843174+0000","last_undegraded":"2023-10-20T10:25:33.843174+0000","last_fullsized":"2023-10-20T10:25:33.843174+0000","mapping_epoch":13,"log_start":"98'8700","ondisk_log_start":"98'8700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":207,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:55.428541+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:24:55.428541+0000","last_peered":"2023-10-20T10:24:55.428541+0000","last_clean":"2023-10-20T10:24:55.428541+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:24:55.428541+0000","last_undegraded":"2023-10-20T10:24:55.428541+0000","last_fullsized":"2023-10-20T10:24:55.428541+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"127'9494","reported_seq":14983,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:25:33.842104+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:25:33.842104+0000","last_peered":"2023-10-20T10:25:33.842104+0000","last_clean":"2023-10-20T10:25:33.842104+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:25:33.842104+0000","last_undegraded":"2023-10-20T10:25:33.842104+0000","last_fullsized":"2023-10-20T10:25:33.842104+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9494,"log_dups_size":0,"ondisk_log_size":9494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"127'486569","reported_seq":965535,"reported_epoch":127,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:25:35.091507+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:25:35.091507+0000","last_peered":"2023-10-20T10:25:35.091507+0000","last_clean":"2023-10-20T10:25:35.091507+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:25:35.091507+0000","last_undegraded":"2023-10-20T10:25:35.091507+0000","last_fullsized":"2023-10-20T10:25:35.091507+0000","mapping_epoch":16,"log_start":"126'476531","ondisk_log_start":"126'476531","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":2430,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 351s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7093325824,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":482050,"num_read_kb":1898444,"num_write":481187,"num_write_kb":1924748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":301,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:54.855620+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:24:54.855620+0000","last_peered":"2023-10-20T10:24:54.855620+0000","last_clean":"2023-10-20T10:24:54.855620+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:24:54.855620+0000","last_undegraded":"2023-10-20T10:24:54.855620+0000","last_fullsized":"2023-10-20T10:24:54.855620+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"127'451825","reported_seq":896267,"reported_epoch":127,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:25:35.690480+0000","last_change":"2023-10-20T10:19:59.154102+0000","last_active":"2023-10-20T10:25:35.690480+0000","last_peered":"2023-10-20T10:25:35.690480+0000","last_clean":"2023-10-20T10:25:35.690480+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:25:35.690480+0000","last_undegraded":"2023-10-20T10:25:35.690480+0000","last_fullsized":"2023-10-20T10:25:35.690480+0000","mapping_epoch":16,"log_start":"126'441741","ondisk_log_start":"126'441741","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3010,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 336s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6576582656,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":447276,"num_read_kb":1759364,"num_write":446232,"num_write_kb":1784928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"126'330","reported_seq":6470,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:25:33.851454+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:25:33.851454+0000","last_peered":"2023-10-20T10:25:33.851454+0000","last_clean":"2023-10-20T10:25:33.851454+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:25:33.851454+0000","last_undegraded":"2023-10-20T10:25:33.851454+0000","last_fullsized":"2023-10-20T10:25:33.851454+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"127'479962","reported_seq":952104,"reported_epoch":127,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:25:35.085894+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:25:35.085894+0000","last_peered":"2023-10-20T10:25:35.085894+0000","last_clean":"2023-10-20T10:25:35.085894+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:25:35.085894+0000","last_undegraded":"2023-10-20T10:25:35.085894+0000","last_fullsized":"2023-10-20T10:25:35.085894+0000","mapping_epoch":16,"log_start":"126'469935","ondisk_log_start":"126'469935","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":2555,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 350s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":6987780096,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":475244,"num_read_kb":1869880,"num_write":474467,"num_write_kb":1897868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"126'307","reported_seq":6319,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:25:33.838947+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:25:33.838947+0000","last_peered":"2023-10-20T10:25:33.838947+0000","last_clean":"2023-10-20T10:25:33.838947+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:25:33.838947+0000","last_undegraded":"2023-10-20T10:25:33.838947+0000","last_fullsized":"2023-10-20T10:25:33.838947+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"127'475909","reported_seq":946565,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:25:35.665077+0000","last_change":"2023-10-20T10:25:34.067677+0000","last_active":"2023-10-20T10:25:35.665077+0000","last_peered":"2023-10-20T10:25:35.665077+0000","last_clean":"2023-10-20T10:25:35.665077+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:25:35.665077+0000","last_undegraded":"2023-10-20T10:25:35.665077+0000","last_fullsized":"2023-10-20T10:25:35.665077+0000","mapping_epoch":16,"log_start":"125'465824","ondisk_log_start":"125'465824","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6951567379,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":473810,"num_read_kb":1864476,"num_write":472435,"num_write_kb":1889734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":206,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:55.428505+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:24:55.428505+0000","last_peered":"2023-10-20T10:24:55.428505+0000","last_clean":"2023-10-20T10:24:55.428505+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:24:55.428505+0000","last_undegraded":"2023-10-20T10:24:55.428505+0000","last_fullsized":"2023-10-20T10:24:55.428505+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":299,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:00.113495+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:24:00.113495+0000","last_peered":"2023-10-20T10:24:00.113495+0000","last_clean":"2023-10-20T10:24:00.113495+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:24:00.113495+0000","last_undegraded":"2023-10-20T10:24:00.113495+0000","last_fullsized":"2023-10-20T10:24:00.113495+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27609255955,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1878380,"num_read_kb":7392164,"num_write":1874321,"num_write_kb":7497278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15353868288,"data_stored":28736622630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40234,"ondisk_log_size":40234,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":29792,"ondisk_log_size":29792,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739235,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9274132,"kb_used_data":7497740,"kb_used_omap":1432,"kb_used_meta":1774951,"kb_avail":85097708,"statfs":{"total":96636764160,"available":87140052992,"internally_reserved":0,"allocated":7677685760,"data_stored":14368209158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466972,"internal_metadata":1817550244},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:25:06 2023","interfaces":[{"interface":"back","average":{"1min":0.28399999999999997,"5min":0.33500000000000002,"15min":0.443},"min":{"1min":0.16800000000000001,"5min":0.16700000000000001,"15min":0.16200000000000001},"max":{"1min":0.498,"5min":1.806,"15min":8.2449999999999992},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.36499999999999999,"15min":0.47599999999999998},"min":{"1min":0.24299999999999999,"5min":0.20499999999999999,"15min":0.18099999999999999},"max":{"1min":0.56100000000000005,"5min":2.6419999999999999,"15min":6.8239999999999998},"last":0.32100000000000001}]},{"osd":1,"last update":"Fri Oct 20 10:25:06 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.443,"15min":0.67700000000000005},"min":{"1min":0.19800000000000001,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":0.55600000000000005,"5min":3.5150000000000001,"15min":35.670000000000002},"last":0.44},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.40000000000000002,"15min":0.56399999999999995},"min":{"1min":0.22900000000000001,"5min":0.22500000000000001,"15min":0.214},"max":{"1min":0.50800000000000001,"5min":2.7090000000000001,"15min":34.948999999999998},"last":0.29199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739235,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5533988,"kb_used_data":3676380,"kb_used_omap":676,"kb_used_meta":1856923,"kb_avail":88837852,"statfs":{"total":96636764160,"available":90969960448,"internally_reserved":0,"allocated":3764613120,"data_stored":7028798586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1901489739},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:24:53 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.45200000000000001,"15min":0.46200000000000002},"min":{"1min":0.19600000000000001,"5min":0.17899999999999999,"15min":0.157},"max":{"1min":1.212,"5min":4.7510000000000003,"15min":4.7510000000000003},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.45300000000000001,"15min":0.46100000000000002},"min":{"1min":0.19800000000000001,"5min":0.186,"15min":0.186},"max":{"1min":5.2560000000000002,"5min":5.4640000000000004,"15min":5.8479999999999999},"last":0.26600000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:24:53 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.40999999999999998,"15min":0.39900000000000002},"min":{"1min":0.222,"5min":0.19900000000000001,"15min":0.18099999999999999},"max":{"1min":0.58099999999999996,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":0.249},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.45800000000000002,"15min":0.44500000000000001},"min":{"1min":0.27400000000000002,"5min":0.252,"15min":0.23799999999999999},"max":{"1min":0.61199999999999999,"5min":4.8109999999999999,"15min":4.8109999999999999},"last":0.36099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739237,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5541124,"kb_used_data":3824444,"kb_used_omap":698,"kb_used_meta":1715973,"kb_avail":88830716,"statfs":{"total":96636764160,"available":90962653184,"internally_reserved":0,"allocated":3916230656,"data_stored":7341395435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":1757156938},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,4,14,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:24:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.70399999999999996,"15min":0.81000000000000005},"min":{"1min":0.182,"5min":0.17799999999999999,"15min":0.16700000000000001},"max":{"1min":2.3290000000000002,"5min":8.0869999999999997,"15min":81.197000000000003},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.54700000000000004,"15min":0.72699999999999998},"min":{"1min":0.22,"5min":0.20000000000000001,"15min":0.184},"max":{"1min":2.363,"5min":6.875,"15min":80.334999999999994},"last":0.23799999999999999}]},{"osd":2,"last update":"Fri Oct 20 10:24:47 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.39700000000000002,"15min":0.504},"min":{"1min":0.216,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":2.3969999999999998,"5min":3.653,"15min":11.176},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.379,"15min":0.54100000000000004},"min":{"1min":0.21199999999999999,"5min":0.20000000000000001,"15min":0.187},"max":{"1min":1.0289999999999999,"5min":1.5049999999999999,"15min":11.093},"last":0.29499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3914358784,"data_stored":7340462080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3762855936,"data_stored":7028129811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7676653568,"data_stored":14368030739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:25:37.402 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:25:37.403 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:25:37.403 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:25:37.404 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:25:37.404 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:25:57.406 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:25:57.905 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:25:57.906 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:25:57.944 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2298,"stamp":"2023-10-20T10:25:57.271915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27639165443,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1923946,"num_read_kb":7452321,"num_write":1882499,"num_write_kb":7528247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70282,"ondisk_log_size":70282,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20581360,"kb_used_data":15055512,"kb_used_omap":2807,"kb_used_meta":5523016,"kb_avail":262534160,"statfs":{"total":289910292480,"available":268834979840,"internally_reserved":0,"allocated":15416844288,"data_stored":28796717931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2874823,"internal_metadata":5655568953},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,3,8,4,11,26],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18485248,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4348,"num_read_kb":17308,"num_write":4518,"num_write_kb":18072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001927"},"pg_stats":[{"pgid":"2.7","version":"127'9580","reported_seq":14306,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:25:54.340080+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:25:54.340080+0000","last_peered":"2023-10-20T10:25:54.340080+0000","last_clean":"2023-10-20T10:25:54.340080+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:25:54.340080+0000","last_undegraded":"2023-10-20T10:25:54.340080+0000","last_fullsized":"2023-10-20T10:25:54.340080+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9580,"log_dups_size":0,"ondisk_log_size":9580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"127'18830","reported_seq":29267,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:25:54.380835+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:25:54.380835+0000","last_peered":"2023-10-20T10:25:54.380835+0000","last_clean":"2023-10-20T10:25:54.380835+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:25:54.380835+0000","last_undegraded":"2023-10-20T10:25:54.380835+0000","last_fullsized":"2023-10-20T10:25:54.380835+0000","mapping_epoch":13,"log_start":"99'8800","ondisk_log_start":"99'8800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":207,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:55.428541+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:24:55.428541+0000","last_peered":"2023-10-20T10:24:55.428541+0000","last_clean":"2023-10-20T10:24:55.428541+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:24:55.428541+0000","last_undegraded":"2023-10-20T10:24:55.428541+0000","last_fullsized":"2023-10-20T10:24:55.428541+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"127'9542","reported_seq":15043,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:25:54.346453+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:25:54.346453+0000","last_peered":"2023-10-20T10:25:54.346453+0000","last_clean":"2023-10-20T10:25:54.346453+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:25:54.346453+0000","last_undegraded":"2023-10-20T10:25:54.346453+0000","last_fullsized":"2023-10-20T10:25:54.346453+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9542,"log_dups_size":0,"ondisk_log_size":9542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"127'488496","reported_seq":969272,"reported_epoch":127,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:25:55.220060+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:25:55.220060+0000","last_peered":"2023-10-20T10:25:55.220060+0000","last_clean":"2023-10-20T10:25:55.220060+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:25:55.220060+0000","last_undegraded":"2023-10-20T10:25:55.220060+0000","last_fullsized":"2023-10-20T10:25:55.220060+0000","mapping_epoch":16,"log_start":"126'478431","ondisk_log_start":"126'478431","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":2536,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 371s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7100915712,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":483850,"num_read_kb":1905600,"num_write":483033,"num_write_kb":1932132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":301,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:54.855620+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:24:54.855620+0000","last_peered":"2023-10-20T10:24:54.855620+0000","last_clean":"2023-10-20T10:24:54.855620+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:24:54.855620+0000","last_undegraded":"2023-10-20T10:24:54.855620+0000","last_fullsized":"2023-10-20T10:24:54.855620+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"127'453725","reported_seq":899845,"reported_epoch":127,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:25:55.777341+0000","last_change":"2023-10-20T10:19:59.154102+0000","last_active":"2023-10-20T10:25:55.777341+0000","last_peered":"2023-10-20T10:25:55.777341+0000","last_clean":"2023-10-20T10:25:55.777341+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:25:55.777341+0000","last_undegraded":"2023-10-20T10:25:55.777341+0000","last_fullsized":"2023-10-20T10:25:55.777341+0000","mapping_epoch":16,"log_start":"126'443641","ondisk_log_start":"126'443641","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3142,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 356s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6583918592,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":448942,"num_read_kb":1766000,"num_write":448030,"num_write_kb":1792120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6488,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:25:54.339837+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:25:54.339837+0000","last_peered":"2023-10-20T10:25:54.339837+0000","last_clean":"2023-10-20T10:25:54.339837+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:25:54.339837+0000","last_undegraded":"2023-10-20T10:25:54.339837+0000","last_fullsized":"2023-10-20T10:25:54.339837+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"127'481814","reported_seq":955641,"reported_epoch":127,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:25:55.171811+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:25:55.171811+0000","last_peered":"2023-10-20T10:25:55.171811+0000","last_clean":"2023-10-20T10:25:55.171811+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:25:55.171811+0000","last_undegraded":"2023-10-20T10:25:55.171811+0000","last_fullsized":"2023-10-20T10:25:55.171811+0000","mapping_epoch":16,"log_start":"126'471735","ondisk_log_start":"126'471735","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":2696,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 370s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":6994911232,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":476916,"num_read_kb":1876540,"num_write":476209,"num_write_kb":1904836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6335,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:25:54.341958+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:25:54.341958+0000","last_peered":"2023-10-20T10:25:54.341958+0000","last_clean":"2023-10-20T10:25:54.341958+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:25:54.341958+0000","last_undegraded":"2023-10-20T10:25:54.341958+0000","last_fullsized":"2023-10-20T10:25:54.341958+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"127'477687","reported_seq":950105,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:25:55.806744+0000","last_change":"2023-10-20T10:25:51.038554+0000","last_active":"2023-10-20T10:25:55.806744+0000","last_peered":"2023-10-20T10:25:55.806744+0000","last_clean":"2023-10-20T10:25:55.806744+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:25:55.806744+0000","last_undegraded":"2023-10-20T10:25:55.806744+0000","last_fullsized":"2023-10-20T10:25:55.806744+0000","mapping_epoch":16,"log_start":"126'467624","ondisk_log_start":"126'467624","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6958825491,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":475567,"num_read_kb":1871460,"num_write":474213,"num_write_kb":1896846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":206,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:55.428505+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:24:55.428505+0000","last_peered":"2023-10-20T10:24:55.428505+0000","last_clean":"2023-10-20T10:24:55.428505+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:24:55.428505+0000","last_undegraded":"2023-10-20T10:24:55.428505+0000","last_fullsized":"2023-10-20T10:24:55.428505+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":299,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-20T10:24:00.113495+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:24:00.113495+0000","last_peered":"2023-10-20T10:24:00.113495+0000","last_clean":"2023-10-20T10:24:00.113495+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:24:00.113495+0000","last_undegraded":"2023-10-20T10:24:00.113495+0000","last_fullsized":"2023-10-20T10:24:00.113495+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27638571027,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1885275,"num_read_kb":7419600,"num_write":1881485,"num_write_kb":7525934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15412527104,"data_stored":28795281446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40291,"ondisk_log_size":40291,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":29892,"ondisk_log_size":29892,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739239,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9380740,"kb_used_data":7526140,"kb_used_omap":1432,"kb_used_meta":1853159,"kb_avail":84991100,"statfs":{"total":96636764160,"available":87030886400,"internally_reserved":0,"allocated":7706767360,"data_stored":14397290758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466972,"internal_metadata":1897635236},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:25:06 2023","interfaces":[{"interface":"back","average":{"1min":0.28399999999999997,"5min":0.33500000000000002,"15min":0.443},"min":{"1min":0.16800000000000001,"5min":0.16700000000000001,"15min":0.16200000000000001},"max":{"1min":0.498,"5min":1.806,"15min":8.2449999999999992},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.36499999999999999,"15min":0.47599999999999998},"min":{"1min":0.24299999999999999,"5min":0.20499999999999999,"15min":0.18099999999999999},"max":{"1min":0.56100000000000005,"5min":2.6419999999999999,"15min":6.8239999999999998},"last":0.51500000000000001}]},{"osd":1,"last update":"Fri Oct 20 10:25:06 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.443,"15min":0.67700000000000005},"min":{"1min":0.19800000000000001,"5min":0.19500000000000001,"15min":0.187},"max":{"1min":0.55600000000000005,"5min":3.5150000000000001,"15min":35.670000000000002},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.40000000000000002,"15min":0.56399999999999995},"min":{"1min":0.22900000000000001,"5min":0.22500000000000001,"15min":0.214},"max":{"1min":0.50800000000000001,"5min":2.7090000000000001,"15min":34.948999999999998},"last":0.46800000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739239,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5584288,"kb_used_data":3690584,"kb_used_omap":676,"kb_used_meta":1893019,"kb_avail":88787552,"statfs":{"total":96636764160,"available":90918453248,"internally_reserved":0,"allocated":3779158016,"data_stored":7043343482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1938452043},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,5,3,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.45000000000000001,"15min":0.44900000000000001},"min":{"1min":0.20000000000000001,"5min":0.17899999999999999,"15min":0.157},"max":{"1min":1.2290000000000001,"5min":4.7510000000000003,"15min":4.7510000000000003},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.45600000000000002,"15min":0.46200000000000002},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.59199999999999997,"5min":5.4640000000000004,"15min":5.8479999999999999},"last":0.216}]},{"osd":2,"last update":"Fri Oct 20 10:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.47099999999999997,"15min":0.40600000000000003},"min":{"1min":0.23000000000000001,"5min":0.217,"15min":0.18099999999999999},"max":{"1min":3.0739999999999998,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":0.309},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.51600000000000001,"15min":0.45100000000000001},"min":{"1min":0.28399999999999997,"5min":0.26400000000000001,"15min":0.23799999999999999},"max":{"1min":3.109,"5min":4.8109999999999999,"15min":4.8109999999999999},"last":0.35899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739241,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5616332,"kb_used_data":3838788,"kb_used_omap":698,"kb_used_meta":1776837,"kb_avail":88755508,"statfs":{"total":96636764160,"available":90885640192,"internally_reserved":0,"allocated":3930918912,"data_stored":7356083691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":1819481674},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,1,4,23],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:25:49 2023","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.65500000000000003,"15min":0.80700000000000005},"min":{"1min":0.188,"5min":0.17799999999999999,"15min":0.16700000000000001},"max":{"1min":0.45700000000000002,"5min":8.0869999999999997,"15min":81.197000000000003},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.315,"5min":0.51500000000000001,"15min":0.72799999999999998},"min":{"1min":0.222,"5min":0.20000000000000001,"15min":0.184},"max":{"1min":0.436,"5min":6.875,"15min":80.334999999999994},"last":0.26200000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:25:49 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.36499999999999999,"15min":0.497},"min":{"1min":0.27500000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.45800000000000002,"5min":2.3969999999999998,"15min":11.176},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38200000000000001,"15min":0.53100000000000003},"min":{"1min":0.23200000000000001,"5min":0.20000000000000001,"15min":0.187},"max":{"1min":0.64800000000000002,"5min":1.5049999999999999,"15min":11.093},"last":0.32200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3929075712,"data_stored":7355179008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3777515520,"data_stored":7042789395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7705935872,"data_stored":14397313043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:25:57.946 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:25:57.947 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:25:57.947 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:25:57.948 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:25:57.948 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:26:17.949 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:26:18.501 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:26:18.501 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:26:18.533 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2309,"stamp":"2023-10-20T10:26:17.275311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27668320771,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1930593,"num_read_kb":7478821,"num_write":1889605,"num_write_kb":7556671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70366,"ondisk_log_size":70366,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20428360,"kb_used_data":15112168,"kb_used_omap":2807,"kb_used_meta":5313352,"kb_avail":262687160,"statfs":{"total":289910292480,"available":268991651840,"internally_reserved":0,"allocated":15474860032,"data_stored":28854710377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2874823,"internal_metadata":5440873017},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,6,3,13,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14942208,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3417,"num_read_kb":13628,"num_write":3640,"num_write_kb":14560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001636"},"pg_stats":[{"pgid":"2.7","version":"128'9613","reported_seq":14352,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:26:14.880855+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:26:14.880855+0000","last_peered":"2023-10-20T10:26:14.880855+0000","last_clean":"2023-10-20T10:26:14.880855+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:26:14.880855+0000","last_undegraded":"2023-10-20T10:26:14.880855+0000","last_fullsized":"2023-10-20T10:26:14.880855+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9613,"log_dups_size":0,"ondisk_log_size":9613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"128'18896","reported_seq":29358,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:26:14.890252+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:26:14.890252+0000","last_peered":"2023-10-20T10:26:14.890252+0000","last_clean":"2023-10-20T10:26:14.890252+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:26:14.890252+0000","last_undegraded":"2023-10-20T10:26:14.890252+0000","last_fullsized":"2023-10-20T10:26:14.890252+0000","mapping_epoch":13,"log_start":"99'8800","ondisk_log_start":"99'8800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":208,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:25:03.119289+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:25:03.119289+0000","last_peered":"2023-10-20T10:25:03.119289+0000","last_clean":"2023-10-20T10:25:03.119289+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:25:03.119289+0000","last_undegraded":"2023-10-20T10:25:03.119289+0000","last_fullsized":"2023-10-20T10:25:03.119289+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"128'9575","reported_seq":15089,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:26:14.872941+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:26:14.872941+0000","last_peered":"2023-10-20T10:26:14.872941+0000","last_clean":"2023-10-20T10:26:14.872941+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:26:14.872941+0000","last_undegraded":"2023-10-20T10:26:14.872941+0000","last_fullsized":"2023-10-20T10:26:14.872941+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9575,"log_dups_size":0,"ondisk_log_size":9575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"128'490427","reported_seq":972974,"reported_epoch":128,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:26:15.385331+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:26:15.385331+0000","last_peered":"2023-10-20T10:26:15.385331+0000","last_clean":"2023-10-20T10:26:15.385331+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:26:15.385331+0000","last_undegraded":"2023-10-20T10:26:15.385331+0000","last_fullsized":"2023-10-20T10:26:15.385331+0000","mapping_epoch":16,"log_start":"126'480331","ondisk_log_start":"126'480331","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":2675,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 391s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7108403200,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":485608,"num_read_kb":1912624,"num_write":484859,"num_write_kb":1939436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":302,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:25:03.111419+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:25:03.111419+0000","last_peered":"2023-10-20T10:25:03.111419+0000","last_clean":"2023-10-20T10:25:03.111419+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:25:03.111419+0000","last_undegraded":"2023-10-20T10:25:03.111419+0000","last_fullsized":"2023-10-20T10:25:03.111419+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"128'455565","reported_seq":903271,"reported_epoch":128,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:26:15.869381+0000","last_change":"2023-10-20T10:19:59.154102+0000","last_active":"2023-10-20T10:26:15.869381+0000","last_peered":"2023-10-20T10:26:15.869381+0000","last_clean":"2023-10-20T10:26:15.869381+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:26:15.869381+0000","last_undegraded":"2023-10-20T10:26:15.869381+0000","last_fullsized":"2023-10-20T10:26:15.869381+0000","mapping_epoch":16,"log_start":"126'445541","ondisk_log_start":"126'445541","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3307,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 376s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6590939136,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":450510,"num_read_kb":1772240,"num_write":449740,"num_write_kb":1798960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6505,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:26:14.825677+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:26:14.825677+0000","last_peered":"2023-10-20T10:26:14.825677+0000","last_clean":"2023-10-20T10:26:14.825677+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:26:14.825677+0000","last_undegraded":"2023-10-20T10:26:14.825677+0000","last_fullsized":"2023-10-20T10:26:14.825677+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"128'483716","reported_seq":959232,"reported_epoch":128,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:26:15.348673+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:26:15.348673+0000","last_peered":"2023-10-20T10:26:15.348673+0000","last_clean":"2023-10-20T10:26:15.348673+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:26:15.348673+0000","last_undegraded":"2023-10-20T10:26:15.348673+0000","last_fullsized":"2023-10-20T10:26:15.348673+0000","mapping_epoch":16,"log_start":"126'473635","ondisk_log_start":"126'473635","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":2843,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 390s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7002243072,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":478593,"num_read_kb":1883224,"num_write":478000,"num_write_kb":1912000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6353,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:26:14.806177+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:26:14.806177+0000","last_peered":"2023-10-20T10:26:14.806177+0000","last_clean":"2023-10-20T10:26:14.806177+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:26:14.806177+0000","last_undegraded":"2023-10-20T10:26:14.806177+0000","last_fullsized":"2023-10-20T10:26:14.806177+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"128'479466","reported_seq":953537,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:26:15.883924+0000","last_change":"2023-10-20T10:26:15.158314+0000","last_active":"2023-10-20T10:26:15.883924+0000","last_peered":"2023-10-20T10:26:15.883924+0000","last_clean":"2023-10-20T10:26:15.883924+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:26:15.883924+0000","last_undegraded":"2023-10-20T10:26:15.883924+0000","last_fullsized":"2023-10-20T10:26:15.883924+0000","mapping_epoch":16,"log_start":"126'469424","ondisk_log_start":"126'469424","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6966140947,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":477211,"num_read_kb":1878012,"num_write":475992,"num_write_kb":1903962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":207,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:25:03.118524+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:25:03.118524+0000","last_peered":"2023-10-20T10:25:03.118524+0000","last_clean":"2023-10-20T10:25:03.118524+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:25:03.118524+0000","last_undegraded":"2023-10-20T10:25:03.118524+0000","last_fullsized":"2023-10-20T10:25:03.118524+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":301,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:26:00.841759+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:26:00.841759+0000","last_peered":"2023-10-20T10:26:00.841759+0000","last_clean":"2023-10-20T10:26:00.841759+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:26:00.841759+0000","last_undegraded":"2023-10-20T10:26:00.841759+0000","last_fullsized":"2023-10-20T10:26:00.841759+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27667726355,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1891922,"num_read_kb":7446100,"num_write":1888591,"num_write_kb":7554358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15470792704,"data_stored":28853547046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40243,"ondisk_log_size":40243,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":30024,"ondisk_log_size":30024,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739243,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9521820,"kb_used_data":7554384,"kb_used_omap":1432,"kb_used_meta":1965991,"kb_avail":84850020,"statfs":{"total":96636764160,"available":86886420480,"internally_reserved":0,"allocated":7735689216,"data_stored":14426204848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466972,"internal_metadata":2013175204},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:26:07 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.38400000000000001,"15min":0.45900000000000002},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16200000000000001},"max":{"1min":4.3799999999999999,"5min":4.3799999999999999,"15min":8.2449999999999992},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.40799999999999997,"15min":0.47299999999999998},"min":{"1min":0.20799999999999999,"5min":0.20499999999999999,"15min":0.18099999999999999},"max":{"1min":4.4850000000000003,"5min":4.4850000000000003,"15min":6.8239999999999998},"last":0.45100000000000001}]},{"osd":1,"last update":"Fri Oct 20 10:26:07 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.45100000000000001,"15min":0.66100000000000003},"min":{"1min":0.22500000000000001,"5min":0.19700000000000001,"15min":0.19500000000000001},"max":{"1min":0.83699999999999997,"5min":3.5150000000000001,"15min":35.670000000000002},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.377,"5min":0.38900000000000001,"15min":0.55000000000000004},"min":{"1min":0.23999999999999999,"5min":0.22800000000000001,"15min":0.214},"max":{"1min":0.81000000000000005,"5min":2.7090000000000001,"15min":34.948999999999998},"last":0.48899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739243,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5243916,"kb_used_data":3704640,"kb_used_omap":676,"kb_used_meta":1538587,"kb_avail":89127924,"statfs":{"total":96636764160,"available":91266994176,"internally_reserved":0,"allocated":3793551360,"data_stored":7057729060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1575513675},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,3,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.45000000000000001,"15min":0.44900000000000001},"min":{"1min":0.20000000000000001,"5min":0.17899999999999999,"15min":0.157},"max":{"1min":1.2290000000000001,"5min":4.7510000000000003,"15min":4.7510000000000003},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.45600000000000002,"15min":0.46200000000000002},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.59199999999999997,"5min":5.4640000000000004,"15min":5.8479999999999999},"last":0.26900000000000002}]},{"osd":2,"last update":"Fri Oct 20 10:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.47099999999999997,"15min":0.40600000000000003},"min":{"1min":0.23000000000000001,"5min":0.217,"15min":0.18099999999999999},"max":{"1min":3.0739999999999998,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":0.44},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.51600000000000001,"15min":0.45100000000000001},"min":{"1min":0.28399999999999997,"5min":0.26400000000000001,"15min":0.23799999999999999},"max":{"1min":3.109,"5min":4.8109999999999999,"15min":4.8109999999999999},"last":0.47499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739245,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5662624,"kb_used_data":3853144,"kb_used_omap":698,"kb_used_meta":1808773,"kb_avail":88709216,"statfs":{"total":96636764160,"available":90838237184,"internally_reserved":0,"allocated":3945619456,"data_stored":7370776469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":1852184138},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,2,10,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:25:49 2023","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.65500000000000003,"15min":0.80700000000000005},"min":{"1min":0.188,"5min":0.17799999999999999,"15min":0.16700000000000001},"max":{"1min":0.45700000000000002,"5min":8.0869999999999997,"15min":81.197000000000003},"last":0.307},{"interface":"front","average":{"1min":0.315,"5min":0.51500000000000001,"15min":0.72799999999999998},"min":{"1min":0.222,"5min":0.20000000000000001,"15min":0.184},"max":{"1min":0.436,"5min":6.875,"15min":80.334999999999994},"last":0.33900000000000002}]},{"osd":2,"last update":"Fri Oct 20 10:25:49 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.36499999999999999,"15min":0.497},"min":{"1min":0.27500000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.45800000000000002,"5min":2.3969999999999998,"15min":11.176},"last":0.38},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38200000000000001,"15min":0.53100000000000003},"min":{"1min":0.23200000000000001,"5min":0.20000000000000001,"15min":0.187},"max":{"1min":0.64800000000000002,"5min":1.5049999999999999,"15min":11.093},"last":0.39600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3943890944,"data_stored":7369994240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3791802368,"data_stored":7057076243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7735099392,"data_stored":14426476563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:26:18.535 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:26:18.535 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:26:18.535 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:26:18.536 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:26:18.536 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-20T10:26:19.096 INFO:teuthology.orchestra.run.smithi150.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-20T10:26:19.147 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-20T10:26:19.668 INFO:teuthology.orchestra.run.smithi150.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-20T10:26:19.708 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-20T10:26:20.334 INFO:teuthology.orchestra.run.smithi150.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-20T10:26:20.394 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-20T10:26:20.918 INFO:teuthology.orchestra.run.smithi150.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-20T10:26:20.950 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:26:40.951 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:26:41.510 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:26:41.511 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:26:41.541 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2321,"stamp":"2023-10-20T10:26:41.279331+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27705246211,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1939057,"num_read_kb":7512557,"num_write":1898606,"num_write_kb":7592675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70334,"ondisk_log_size":70334,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20745588,"kb_used_data":15184088,"kb_used_omap":2807,"kb_used_meta":5558664,"kb_avail":262369932,"statfs":{"total":289910292480,"available":268666810368,"internally_reserved":0,"allocated":15548506112,"data_stored":28928356457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2874823,"internal_metadata":5692072505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,5,6,4,26,16],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22401024,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5170,"num_read_kb":20616,"num_write":5459,"num_write_kb":21836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002050"},"pg_stats":[{"pgid":"2.7","version":"128'9661","reported_seq":14415,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:26:40.398114+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:26:40.398114+0000","last_peered":"2023-10-20T10:26:40.398114+0000","last_clean":"2023-10-20T10:26:40.398114+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:26:40.398114+0000","last_undegraded":"2023-10-20T10:26:40.398114+0000","last_fullsized":"2023-10-20T10:26:40.398114+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9661,"log_dups_size":0,"ondisk_log_size":9661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"128'18992","reported_seq":29484,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:26:40.399263+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:26:40.399263+0000","last_peered":"2023-10-20T10:26:40.399263+0000","last_clean":"2023-10-20T10:26:40.399263+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:26:40.399263+0000","last_undegraded":"2023-10-20T10:26:40.399263+0000","last_fullsized":"2023-10-20T10:26:40.399263+0000","mapping_epoch":13,"log_start":"100'8900","ondisk_log_start":"100'8900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":208,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:25:03.119289+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:25:03.119289+0000","last_peered":"2023-10-20T10:25:03.119289+0000","last_clean":"2023-10-20T10:25:03.119289+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:25:03.119289+0000","last_undegraded":"2023-10-20T10:25:03.119289+0000","last_fullsized":"2023-10-20T10:25:03.119289+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"128'9623","reported_seq":15152,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:26:40.398321+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:26:40.398321+0000","last_peered":"2023-10-20T10:26:40.398321+0000","last_clean":"2023-10-20T10:26:40.398321+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:26:40.398321+0000","last_undegraded":"2023-10-20T10:26:40.398321+0000","last_fullsized":"2023-10-20T10:26:40.398321+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9623,"log_dups_size":0,"ondisk_log_size":9623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"128'492850","reported_seq":977578,"reported_epoch":128,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:26:40.573098+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:26:40.573098+0000","last_peered":"2023-10-20T10:26:40.573098+0000","last_clean":"2023-10-20T10:26:40.573098+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:26:40.573098+0000","last_undegraded":"2023-10-20T10:26:40.573098+0000","last_fullsized":"2023-10-20T10:26:40.573098+0000","mapping_epoch":16,"log_start":"126'482831","ondisk_log_start":"126'482831","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":2815,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 416s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7117881344,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":487774,"num_read_kb":1921280,"num_write":487171,"num_write_kb":1948684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":302,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:25:03.111419+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:25:03.111419+0000","last_peered":"2023-10-20T10:25:03.111419+0000","last_clean":"2023-10-20T10:25:03.111419+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:25:03.111419+0000","last_undegraded":"2023-10-20T10:25:03.111419+0000","last_fullsized":"2023-10-20T10:25:03.111419+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"128'457850","reported_seq":907599,"reported_epoch":128,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:26:41.008457+0000","last_change":"2023-10-20T10:19:59.154102+0000","last_active":"2023-10-20T10:26:41.008457+0000","last_peered":"2023-10-20T10:26:41.008457+0000","last_clean":"2023-10-20T10:26:41.008457+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:26:41.008457+0000","last_undegraded":"2023-10-20T10:26:41.008457+0000","last_fullsized":"2023-10-20T10:26:41.008457+0000","mapping_epoch":16,"log_start":"126'447841","ondisk_log_start":"126'447841","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3495,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 401s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6599753728,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":452540,"num_read_kb":1780332,"num_write":451881,"num_write_kb":1807524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6521,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:26:35.333534+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:26:35.333534+0000","last_peered":"2023-10-20T10:26:35.333534+0000","last_clean":"2023-10-20T10:26:35.333534+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:26:35.333534+0000","last_undegraded":"2023-10-20T10:26:35.333534+0000","last_fullsized":"2023-10-20T10:26:35.333534+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"128'486094","reported_seq":963789,"reported_epoch":128,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:26:40.573161+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:26:40.573161+0000","last_peered":"2023-10-20T10:26:40.573161+0000","last_clean":"2023-10-20T10:26:40.573161+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:26:40.573161+0000","last_undegraded":"2023-10-20T10:26:40.573161+0000","last_fullsized":"2023-10-20T10:26:40.573161+0000","mapping_epoch":16,"log_start":"126'476035","ondisk_log_start":"126'476035","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3004,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 415s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7011483648,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":480754,"num_read_kb":1891812,"num_write":480258,"num_write_kb":1921032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6369,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:26:35.280963+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:26:35.280963+0000","last_peered":"2023-10-20T10:26:35.280963+0000","last_clean":"2023-10-20T10:26:35.280963+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:26:35.280963+0000","last_undegraded":"2023-10-20T10:26:35.280963+0000","last_fullsized":"2023-10-20T10:26:35.280963+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"128'481756","reported_seq":957941,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:26:41.018972+0000","last_change":"2023-10-20T10:26:37.441398+0000","last_active":"2023-10-20T10:26:41.018972+0000","last_peered":"2023-10-20T10:26:41.018972+0000","last_clean":"2023-10-20T10:26:41.018972+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:26:41.018972+0000","last_undegraded":"2023-10-20T10:26:41.018972+0000","last_fullsized":"2023-10-20T10:26:41.018972+0000","mapping_epoch":16,"log_start":"126'471724","ondisk_log_start":"126'471724","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6975533075,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":479318,"num_read_kb":1886412,"num_write":478282,"num_write_kb":1913122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":207,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:25:03.118524+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:25:03.118524+0000","last_peered":"2023-10-20T10:25:03.118524+0000","last_clean":"2023-10-20T10:25:03.118524+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:25:03.118524+0000","last_undegraded":"2023-10-20T10:25:03.118524+0000","last_fullsized":"2023-10-20T10:25:03.118524+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":301,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:26:00.841759+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:26:00.841759+0000","last_peered":"2023-10-20T10:26:00.841759+0000","last_clean":"2023-10-20T10:26:00.841759+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:26:00.841759+0000","last_undegraded":"2023-10-20T10:26:00.841759+0000","last_fullsized":"2023-10-20T10:26:00.841759+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27704651795,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1900386,"num_read_kb":7479836,"num_write":1897592,"num_write_kb":7590362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15544430592,"data_stored":28927184934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40119,"ondisk_log_size":40119,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":30116,"ondisk_log_size":30116,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739248,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9684000,"kb_used_data":7590228,"kb_used_omap":1432,"kb_used_meta":2092327,"kb_avail":84687840,"statfs":{"total":96636764160,"available":86720348160,"internally_reserved":0,"allocated":7772393472,"data_stored":14462909104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1466972,"internal_metadata":2142543268},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:26:07 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.38400000000000001,"15min":0.45900000000000002},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16200000000000001},"max":{"1min":4.3799999999999999,"5min":4.3799999999999999,"15min":8.2449999999999992},"last":0.192},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.40799999999999997,"15min":0.47299999999999998},"min":{"1min":0.20799999999999999,"5min":0.20499999999999999,"15min":0.18099999999999999},"max":{"1min":4.4850000000000003,"5min":4.4850000000000003,"15min":6.8239999999999998},"last":0.25800000000000001}]},{"osd":1,"last update":"Fri Oct 20 10:26:07 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.45100000000000001,"15min":0.66100000000000003},"min":{"1min":0.22500000000000001,"5min":0.19700000000000001,"15min":0.19500000000000001},"max":{"1min":0.83699999999999997,"5min":3.5150000000000001,"15min":35.670000000000002},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.377,"5min":0.38900000000000001,"15min":0.55000000000000004},"min":{"1min":0.23999999999999999,"5min":0.22800000000000001,"15min":0.214},"max":{"1min":0.81000000000000005,"5min":2.7090000000000001,"15min":34.948999999999998},"last":0.23400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739248,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5304308,"kb_used_data":3722412,"kb_used_omap":676,"kb_used_meta":1581211,"kb_avail":89067532,"statfs":{"total":96636764160,"available":91205152768,"internally_reserved":0,"allocated":3811749888,"data_stored":7075927588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1619160651},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,0,1,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.45000000000000001,"15min":0.44900000000000001},"min":{"1min":0.20000000000000001,"5min":0.17899999999999999,"15min":0.157},"max":{"1min":1.2290000000000001,"5min":4.7510000000000003,"15min":4.7510000000000003},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.45600000000000002,"15min":0.46200000000000002},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.59199999999999997,"5min":5.4640000000000004,"15min":5.8479999999999999},"last":0.34399999999999997}]},{"osd":2,"last update":"Fri Oct 20 10:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.47099999999999997,"15min":0.40600000000000003},"min":{"1min":0.23000000000000001,"5min":0.217,"15min":0.18099999999999999},"max":{"1min":3.0739999999999998,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.51600000000000001,"15min":0.45100000000000001},"min":{"1min":0.28399999999999997,"5min":0.26400000000000001,"15min":0.23799999999999999},"max":{"1min":3.109,"5min":4.8109999999999999,"15min":4.8109999999999999},"last":0.61299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739250,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5757280,"kb_used_data":3871448,"kb_used_omap":698,"kb_used_meta":1885125,"kb_avail":88614560,"statfs":{"total":96636764160,"available":90741309440,"internally_reserved":0,"allocated":3964362752,"data_stored":7389519765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":1930368586},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,3,1,15,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:25:49 2023","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.65500000000000003,"15min":0.80700000000000005},"min":{"1min":0.188,"5min":0.17799999999999999,"15min":0.16700000000000001},"max":{"1min":0.45700000000000002,"5min":8.0869999999999997,"15min":81.197000000000003},"last":3.1539999999999999},{"interface":"front","average":{"1min":0.315,"5min":0.51500000000000001,"15min":0.72799999999999998},"min":{"1min":0.222,"5min":0.20000000000000001,"15min":0.184},"max":{"1min":0.436,"5min":6.875,"15min":80.334999999999994},"last":0.25700000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:25:49 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.36499999999999999,"15min":0.497},"min":{"1min":0.27500000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.45800000000000002,"5min":2.3969999999999998,"15min":11.176},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38200000000000001,"15min":0.53100000000000003},"min":{"1min":0.23200000000000001,"5min":0.20000000000000001,"15min":0.187},"max":{"1min":0.64800000000000002,"5min":1.5049999999999999,"15min":11.093},"last":0.318}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3962597376,"data_stored":7388700672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3809951744,"data_stored":7075225619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7771881472,"data_stored":14463258643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:26:41.543 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:26:41.544 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:26:41.544 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:26:41.544 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:26:41.545 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:27:01.551 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:27:02.051 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:27:02.052 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:27:02.080 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2331,"stamp":"2023-10-20T10:27:01.282622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27734761987,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1945800,"num_read_kb":7539477,"num_write":1905781,"num_write_kb":7621375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70459,"ondisk_log_size":70459,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20321788,"kb_used_data":15241444,"kb_used_omap":3120,"kb_used_meta":5077199,"kb_avail":262793732,"statfs":{"total":289910292480,"available":269100781568,"internally_reserved":0,"allocated":15607238656,"data_stored":28987089001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3195282,"internal_metadata":5199052398},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,9,2,6,15,10,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21979136,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5061,"num_read_kb":20204,"num_write":5349,"num_write_kb":21396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001952"},"pg_stats":[{"pgid":"2.7","version":"128'9691","reported_seq":14454,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:27:00.877979+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:27:00.877979+0000","last_peered":"2023-10-20T10:27:00.877979+0000","last_clean":"2023-10-20T10:27:00.877979+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:27:00.877979+0000","last_undegraded":"2023-10-20T10:27:00.877979+0000","last_fullsized":"2023-10-20T10:27:00.877979+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9691,"log_dups_size":0,"ondisk_log_size":9691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"128'19052","reported_seq":29562,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:27:00.878623+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:27:00.878623+0000","last_peered":"2023-10-20T10:27:00.878623+0000","last_clean":"2023-10-20T10:27:00.878623+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:27:00.878623+0000","last_undegraded":"2023-10-20T10:27:00.878623+0000","last_fullsized":"2023-10-20T10:27:00.878623+0000","mapping_epoch":13,"log_start":"100'9000","ondisk_log_start":"100'9000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":209,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:26:13.141126+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:26:13.141126+0000","last_peered":"2023-10-20T10:26:13.141126+0000","last_clean":"2023-10-20T10:26:13.141126+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:26:13.141126+0000","last_undegraded":"2023-10-20T10:26:13.141126+0000","last_fullsized":"2023-10-20T10:26:13.141126+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"128'9653","reported_seq":15192,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:27:00.878243+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:27:00.878243+0000","last_peered":"2023-10-20T10:27:00.878243+0000","last_clean":"2023-10-20T10:27:00.878243+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:27:00.878243+0000","last_undegraded":"2023-10-20T10:27:00.878243+0000","last_fullsized":"2023-10-20T10:27:00.878243+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9653,"log_dups_size":0,"ondisk_log_size":9653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"128'494787","reported_seq":981248,"reported_epoch":128,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:27:00.699994+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:27:00.699994+0000","last_peered":"2023-10-20T10:27:00.699994+0000","last_clean":"2023-10-20T10:27:00.699994+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:27:00.699994+0000","last_undegraded":"2023-10-20T10:27:00.699994+0000","last_fullsized":"2023-10-20T10:27:00.699994+0000","mapping_epoch":16,"log_start":"127'484731","ondisk_log_start":"127'484731","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":2929,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 436s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7125504000,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":489495,"num_read_kb":1928148,"num_write":489023,"num_write_kb":1956092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":303,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:26:13.126146+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:26:13.126146+0000","last_peered":"2023-10-20T10:26:13.126146+0000","last_clean":"2023-10-20T10:26:13.126146+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:26:13.126146+0000","last_undegraded":"2023-10-20T10:26:13.126146+0000","last_fullsized":"2023-10-20T10:26:13.126146+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"128'459644","reported_seq":911001,"reported_epoch":128,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:27:01.130096+0000","last_change":"2023-10-20T10:19:59.154102+0000","last_active":"2023-10-20T10:27:01.130096+0000","last_peered":"2023-10-20T10:27:01.130096+0000","last_clean":"2023-10-20T10:27:01.130096+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:27:01.130096+0000","last_undegraded":"2023-10-20T10:27:01.130096+0000","last_fullsized":"2023-10-20T10:27:01.130096+0000","mapping_epoch":16,"log_start":"127'449642","ondisk_log_start":"127'449642","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3629,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 421s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6606692352,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":454137,"num_read_kb":1786712,"num_write":453572,"num_write_kb":1814288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6538,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:26:55.805529+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:26:55.805529+0000","last_peered":"2023-10-20T10:26:55.805529+0000","last_clean":"2023-10-20T10:26:55.805529+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:26:55.805529+0000","last_undegraded":"2023-10-20T10:26:55.805529+0000","last_fullsized":"2023-10-20T10:26:55.805529+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"128'488035","reported_seq":967459,"reported_epoch":128,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:27:00.699808+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:27:00.699808+0000","last_peered":"2023-10-20T10:27:00.699808+0000","last_clean":"2023-10-20T10:27:00.699808+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:27:00.699808+0000","last_undegraded":"2023-10-20T10:27:00.699808+0000","last_fullsized":"2023-10-20T10:27:00.699808+0000","mapping_epoch":16,"log_start":"127'477935","ondisk_log_start":"127'477935","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3187,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 435s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7018917888,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":482469,"num_read_kb":1898660,"num_write":482056,"num_write_kb":1928224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6385,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:26:55.762449+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:26:55.762449+0000","last_peered":"2023-10-20T10:26:55.762449+0000","last_clean":"2023-10-20T10:26:55.762449+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:26:55.762449+0000","last_undegraded":"2023-10-20T10:26:55.762449+0000","last_fullsized":"2023-10-20T10:26:55.762449+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"128'483590","reported_seq":961496,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:27:01.151184+0000","last_change":"2023-10-20T10:26:58.544990+0000","last_active":"2023-10-20T10:27:01.151184+0000","last_peered":"2023-10-20T10:27:01.151184+0000","last_clean":"2023-10-20T10:27:01.151184+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:27:01.151184+0000","last_undegraded":"2023-10-20T10:27:01.151184+0000","last_fullsized":"2023-10-20T10:27:01.151184+0000","mapping_epoch":16,"log_start":"127'473524","ondisk_log_start":"127'473524","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6983053331,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":481028,"num_read_kb":1893236,"num_write":480116,"num_write_kb":1920458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":208,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:26:13.141082+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:26:13.141082+0000","last_peered":"2023-10-20T10:26:13.141082+0000","last_clean":"2023-10-20T10:26:13.141082+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:26:13.141082+0000","last_undegraded":"2023-10-20T10:26:13.141082+0000","last_fullsized":"2023-10-20T10:26:13.141082+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":301,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:26:00.841759+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:26:00.841759+0000","last_peered":"2023-10-20T10:26:00.841759+0000","last_clean":"2023-10-20T10:26:00.841759+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:26:00.841759+0000","last_undegraded":"2023-10-20T10:26:00.841759+0000","last_fullsized":"2023-10-20T10:26:00.841759+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27734167571,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1907129,"num_read_kb":7506756,"num_write":1904767,"num_write_kb":7619062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15603183616,"data_stored":28985937958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40224,"ondisk_log_size":40224,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":30136,"ondisk_log_size":30136,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739252,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9159396,"kb_used_data":7618908,"kb_used_omap":1745,"kb_used_meta":1538734,"kb_avail":85212444,"statfs":{"total":96636764160,"available":87257542656,"internally_reserved":0,"allocated":7801761792,"data_stored":14492277424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1787431,"internal_metadata":1575664089},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:26:07 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.38400000000000001,"15min":0.45900000000000002},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16200000000000001},"max":{"1min":4.3799999999999999,"5min":4.3799999999999999,"15min":8.2449999999999992},"last":0.245},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.40799999999999997,"15min":0.47299999999999998},"min":{"1min":0.20799999999999999,"5min":0.20499999999999999,"15min":0.18099999999999999},"max":{"1min":4.4850000000000003,"5min":4.4850000000000003,"15min":6.8239999999999998},"last":0.33000000000000002}]},{"osd":1,"last update":"Fri Oct 20 10:26:07 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.45100000000000001,"15min":0.66100000000000003},"min":{"1min":0.22500000000000001,"5min":0.19700000000000001,"15min":0.19500000000000001},"max":{"1min":0.83699999999999997,"5min":3.5150000000000001,"15min":35.670000000000002},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.38900000000000001,"15min":0.55000000000000004},"min":{"1min":0.23999999999999999,"5min":0.22800000000000001,"15min":0.214},"max":{"1min":0.81000000000000005,"5min":2.7090000000000001,"15min":34.948999999999998},"last":0.29999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739252,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5354436,"kb_used_data":3736444,"kb_used_omap":676,"kb_used_meta":1617307,"kb_avail":89017404,"statfs":{"total":96636764160,"available":91153821696,"internally_reserved":0,"allocated":3826118656,"data_stored":7090296356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1656122955},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,2,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.50600000000000001,"15min":0.46800000000000003},"min":{"1min":0.20799999999999999,"5min":0.17899999999999999,"15min":0.157},"max":{"1min":4.9480000000000004,"5min":4.9480000000000004,"15min":4.9480000000000004},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.54800000000000004,"15min":0.498},"min":{"1min":0.20599999999999999,"5min":0.186,"15min":0.186},"max":{"1min":5.0049999999999999,"5min":5.4640000000000004,"15min":5.8479999999999999},"last":0.253}]},{"osd":2,"last update":"Fri Oct 20 10:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.50700000000000001,"15min":0.41399999999999998},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":2.5659999999999998,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.55500000000000005,"15min":0.45900000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.23799999999999999},"max":{"1min":2.6019999999999999,"5min":4.8109999999999999,"15min":4.8109999999999999},"last":0.32700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739254,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5807956,"kb_used_data":3886092,"kb_used_omap":698,"kb_used_meta":1921157,"kb_avail":88563884,"statfs":{"total":96636764160,"available":90689417216,"internally_reserved":0,"allocated":3979358208,"data_stored":7404515221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":1967265354},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,2,10,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:26:52 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.72099999999999997,"15min":0.81499999999999995},"min":{"1min":0.19500000000000001,"5min":0.17799999999999999,"15min":0.16700000000000001},"max":{"1min":4.5609999999999999,"5min":8.0869999999999997,"15min":81.197000000000003},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.28899999999999998,"5min":0.49399999999999999,"15min":0.72199999999999998},"min":{"1min":0.215,"5min":0.21299999999999999,"15min":0.184},"max":{"1min":0.52000000000000002,"5min":6.875,"15min":80.334999999999994},"last":0.33700000000000002}]},{"osd":2,"last update":"Fri Oct 20 10:26:52 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.371,"15min":0.48999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.57199999999999995,"5min":2.3969999999999998,"15min":11.176},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.38400000000000001,"15min":0.52600000000000002},"min":{"1min":0.21099999999999999,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":0.752,"5min":1.093,"15min":11.093},"last":0.35499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3977539584,"data_stored":7403642880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3824390144,"data_stored":7089664019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7801253888,"data_stored":14492631059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:27:02.082 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:27:02.083 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:27:02.083 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:27:02.083 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:27:02.084 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:27:22.085 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:27:22.750 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:27:22.751 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:27:22.802 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2341,"stamp":"2023-10-20T10:27:21.286289+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27764597251,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1952744,"num_read_kb":7567193,"num_write":1913033,"num_write_kb":7650383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70377,"ondisk_log_size":70377,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20537616,"kb_used_data":15299000,"kb_used_omap":3120,"kb_used_meta":5235471,"kb_avail":262577904,"statfs":{"total":289910292480,"available":268879773696,"internally_reserved":0,"allocated":15666176000,"data_stored":29046026345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3195282,"internal_metadata":5361122926},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,2,5,6,15,24,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22626304,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5242,"num_read_kb":20920,"num_write":5489,"num_write_kb":21956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002260"},"pg_stats":[{"pgid":"2.7","version":"128'9730","reported_seq":14502,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:27:19.765139+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:27:19.765139+0000","last_peered":"2023-10-20T10:27:19.765139+0000","last_clean":"2023-10-20T10:27:19.765139+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:27:19.765139+0000","last_undegraded":"2023-10-20T10:27:19.765139+0000","last_fullsized":"2023-10-20T10:27:19.765139+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9730,"log_dups_size":0,"ondisk_log_size":9730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"128'19130","reported_seq":29658,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:27:19.781083+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:27:19.781083+0000","last_peered":"2023-10-20T10:27:19.781083+0000","last_clean":"2023-10-20T10:27:19.781083+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:27:19.781083+0000","last_undegraded":"2023-10-20T10:27:19.781083+0000","last_fullsized":"2023-10-20T10:27:19.781083+0000","mapping_epoch":13,"log_start":"101'9100","ondisk_log_start":"101'9100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":209,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:26:13.141126+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:26:13.141126+0000","last_peered":"2023-10-20T10:26:13.141126+0000","last_clean":"2023-10-20T10:26:13.141126+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:26:13.141126+0000","last_undegraded":"2023-10-20T10:26:13.141126+0000","last_fullsized":"2023-10-20T10:26:13.141126+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"128'9692","reported_seq":15240,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:27:19.764274+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:27:19.764274+0000","last_peered":"2023-10-20T10:27:19.764274+0000","last_clean":"2023-10-20T10:27:19.764274+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:27:19.764274+0000","last_undegraded":"2023-10-20T10:27:19.764274+0000","last_fullsized":"2023-10-20T10:27:19.764274+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9692,"log_dups_size":0,"ondisk_log_size":9692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"128'496765","reported_seq":984985,"reported_epoch":128,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:27:20.838082+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:27:20.838082+0000","last_peered":"2023-10-20T10:27:20.838082+0000","last_clean":"2023-10-20T10:27:20.838082+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:27:20.838082+0000","last_undegraded":"2023-10-20T10:27:20.838082+0000","last_fullsized":"2023-10-20T10:27:20.838082+0000","mapping_epoch":16,"log_start":"127'486731","ondisk_log_start":"127'486731","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3026,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 456s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7133335552,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":491246,"num_read_kb":1935144,"num_write":490929,"num_write_kb":1963716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":303,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:26:13.126146+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:26:13.126146+0000","last_peered":"2023-10-20T10:26:13.126146+0000","last_clean":"2023-10-20T10:26:13.126146+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:26:13.126146+0000","last_undegraded":"2023-10-20T10:26:13.126146+0000","last_fullsized":"2023-10-20T10:26:13.126146+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"128'461543","reported_seq":914581,"reported_epoch":128,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:27:21.260595+0000","last_change":"2023-10-20T10:19:59.154102+0000","last_active":"2023-10-20T10:27:21.260595+0000","last_peered":"2023-10-20T10:27:21.260595+0000","last_clean":"2023-10-20T10:27:21.260595+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:27:21.260595+0000","last_undegraded":"2023-10-20T10:27:21.260595+0000","last_fullsized":"2023-10-20T10:27:21.260595+0000","mapping_epoch":16,"log_start":"127'451542","ondisk_log_start":"127'451542","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3801,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 442s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6613946368,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":455804,"num_read_kb":1793360,"num_write":455336,"num_write_kb":1821344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6554,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:27:16.264114+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:27:16.264114+0000","last_peered":"2023-10-20T10:27:16.264114+0000","last_clean":"2023-10-20T10:27:16.264114+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:27:16.264114+0000","last_undegraded":"2023-10-20T10:27:16.264114+0000","last_fullsized":"2023-10-20T10:27:16.264114+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"128'489959","reported_seq":971126,"reported_epoch":128,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:27:20.837526+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:27:20.837526+0000","last_peered":"2023-10-20T10:27:20.837526+0000","last_clean":"2023-10-20T10:27:20.837526+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:27:20.837526+0000","last_undegraded":"2023-10-20T10:27:20.837526+0000","last_fullsized":"2023-10-20T10:27:20.837526+0000","mapping_epoch":16,"log_start":"127'479935","ondisk_log_start":"127'479935","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3319,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 455s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7026438144,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":484202,"num_read_kb":1905572,"num_write":483877,"num_write_kb":1935508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6401,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:27:16.238686+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:27:16.238686+0000","last_peered":"2023-10-20T10:27:16.238686+0000","last_clean":"2023-10-20T10:27:16.238686+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:27:16.238686+0000","last_undegraded":"2023-10-20T10:27:16.238686+0000","last_fullsized":"2023-10-20T10:27:16.238686+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"128'485351","reported_seq":965056,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:27:21.260747+0000","last_change":"2023-10-20T10:27:20.667856+0000","last_active":"2023-10-20T10:27:21.260747+0000","last_peered":"2023-10-20T10:27:21.260747+0000","last_clean":"2023-10-20T10:27:21.260747+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:27:21.260747+0000","last_undegraded":"2023-10-20T10:27:21.260747+0000","last_fullsized":"2023-10-20T10:27:21.260747+0000","mapping_epoch":16,"log_start":"127'475324","ondisk_log_start":"127'475324","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6990282771,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":482821,"num_read_kb":1900396,"num_write":481877,"num_write_kb":1927502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":208,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:26:13.141082+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:26:13.141082+0000","last_peered":"2023-10-20T10:26:13.141082+0000","last_clean":"2023-10-20T10:26:13.141082+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:26:13.141082+0000","last_undegraded":"2023-10-20T10:26:13.141082+0000","last_fullsized":"2023-10-20T10:26:13.141082+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":301,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-20T10:26:00.841759+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:26:00.841759+0000","last_peered":"2023-10-20T10:26:00.841759+0000","last_clean":"2023-10-20T10:26:00.841759+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:26:00.841759+0000","last_undegraded":"2023-10-20T10:26:00.841759+0000","last_fullsized":"2023-10-20T10:26:00.841759+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27764002835,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1914073,"num_read_kb":7534472,"num_write":1912019,"num_write_kb":7648070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15662555136,"data_stored":29045309478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40086,"ondisk_log_size":40086,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":30192,"ondisk_log_size":30192,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739256,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9274148,"kb_used_data":7647580,"kb_used_omap":1745,"kb_used_meta":1624814,"kb_avail":85097692,"statfs":{"total":96636764160,"available":87140036608,"internally_reserved":0,"allocated":7831121920,"data_stored":14521637552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1787431,"internal_metadata":1663810009},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:27:08 2023","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.378,"15min":0.44500000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":0.55600000000000005,"5min":4.3799999999999999,"15min":8.2449999999999992},"last":2.8919999999999999},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.48699999999999999,"15min":0.48199999999999998},"min":{"1min":0.219,"5min":0.20799999999999999,"15min":0.18099999999999999},"max":{"1min":5.048,"5min":5.048,"15min":6.8239999999999998},"last":2.855}]},{"osd":1,"last update":"Fri Oct 20 10:27:08 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":0.58099999999999996,"15min":0.70799999999999996},"min":{"1min":0.20999999999999999,"5min":0.19700000000000001,"15min":0.19500000000000001},"max":{"1min":6.1509999999999998,"5min":6.1509999999999998,"15min":35.670000000000002},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.46899999999999997,"15min":0.57299999999999995},"min":{"1min":0.23400000000000001,"5min":0.22800000000000001,"15min":0.214},"max":{"1min":4.8769999999999998,"5min":4.8769999999999998,"15min":34.948999999999998},"last":0.26300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739256,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5404672,"kb_used_data":3750584,"kb_used_omap":676,"kb_used_meta":1653403,"kb_avail":88967168,"statfs":{"total":96636764160,"available":91102380032,"internally_reserved":0,"allocated":3840598016,"data_stored":7104775716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1693085259},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,1,3,13,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.50600000000000001,"15min":0.46800000000000003},"min":{"1min":0.20799999999999999,"5min":0.17899999999999999,"15min":0.157},"max":{"1min":4.9480000000000004,"5min":4.9480000000000004,"15min":4.9480000000000004},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.54800000000000004,"15min":0.498},"min":{"1min":0.20599999999999999,"5min":0.186,"15min":0.186},"max":{"1min":5.0049999999999999,"5min":5.4640000000000004,"15min":5.8479999999999999},"last":0.22900000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.50700000000000001,"15min":0.41399999999999998},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":2.5659999999999998,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":0.311},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.55500000000000005,"15min":0.45900000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.23799999999999999},"max":{"1min":2.6019999999999999,"5min":4.8109999999999999,"15min":4.8109999999999999},"last":0.33700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739258,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5858796,"kb_used_data":3900836,"kb_used_omap":698,"kb_used_meta":1957253,"kb_avail":88513044,"statfs":{"total":96636764160,"available":90637357056,"internally_reserved":0,"allocated":3994456064,"data_stored":7419613077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":2004227658},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,2,8,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:26:52 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.72099999999999997,"15min":0.81499999999999995},"min":{"1min":0.19500000000000001,"5min":0.17799999999999999,"15min":0.16700000000000001},"max":{"1min":4.5609999999999999,"5min":8.0869999999999997,"15min":81.197000000000003},"last":0.224},{"interface":"front","average":{"1min":0.28899999999999998,"5min":0.49399999999999999,"15min":0.72199999999999998},"min":{"1min":0.215,"5min":0.21299999999999999,"15min":0.184},"max":{"1min":0.52000000000000002,"5min":6.875,"15min":80.334999999999994},"last":0.25800000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:26:52 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.371,"15min":0.48999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.57199999999999995,"5min":2.3969999999999998,"15min":11.176},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.38400000000000001,"15min":0.52600000000000002},"min":{"1min":0.21099999999999999,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":0.752,"5min":1.093,"15min":11.093},"last":0.38500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3992805376,"data_stored":7418908672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3838820352,"data_stored":7104094227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7830929408,"data_stored":14522306579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:27:22.804 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:27:22.805 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:27:22.805 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:27:22.806 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:27:22.806 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:27:42.808 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:27:43.382 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:27:43.383 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:27:43.427 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2353,"stamp":"2023-10-20T10:27:43.290110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27794067971,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1959749,"num_read_kb":7595157,"num_write":1920227,"num_write_kb":7679159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70441,"ondisk_log_size":70441,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20765696,"kb_used_data":15356392,"kb_used_omap":3120,"kb_used_meta":5406159,"kb_avail":262349824,"statfs":{"total":289910292480,"available":268646219776,"internally_reserved":0,"allocated":15724945408,"data_stored":29104772326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3195282,"internal_metadata":5535907438},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,12,5,26,10,13,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18497536,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4397,"num_read_kb":17556,"num_write":4509,"num_write_kb":18036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002019"},"pg_stats":[{"pgid":"2.7","version":"129'9766","reported_seq":14551,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:27:41.275422+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:27:41.275422+0000","last_peered":"2023-10-20T10:27:41.275422+0000","last_clean":"2023-10-20T10:27:41.275422+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:27:41.275422+0000","last_undegraded":"2023-10-20T10:27:41.275422+0000","last_fullsized":"2023-10-20T10:27:41.275422+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9766,"log_dups_size":0,"ondisk_log_size":9766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"129'19202","reported_seq":29756,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:27:41.296686+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:27:41.296686+0000","last_peered":"2023-10-20T10:27:41.296686+0000","last_clean":"2023-10-20T10:27:41.296686+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:27:41.296686+0000","last_undegraded":"2023-10-20T10:27:41.296686+0000","last_fullsized":"2023-10-20T10:27:41.296686+0000","mapping_epoch":13,"log_start":"104'9200","ondisk_log_start":"104'9200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":210,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:27:01.151399+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:27:01.151399+0000","last_peered":"2023-10-20T10:27:01.151399+0000","last_clean":"2023-10-20T10:27:01.151399+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:27:01.151399+0000","last_undegraded":"2023-10-20T10:27:01.151399+0000","last_fullsized":"2023-10-20T10:27:01.151399+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"129'9728","reported_seq":15289,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:27:41.266573+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:27:41.266573+0000","last_peered":"2023-10-20T10:27:41.266573+0000","last_clean":"2023-10-20T10:27:41.266573+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:27:41.266573+0000","last_undegraded":"2023-10-20T10:27:41.266573+0000","last_fullsized":"2023-10-20T10:27:41.266573+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9728,"log_dups_size":0,"ondisk_log_size":9728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"129'498688","reported_seq":988734,"reported_epoch":129,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:27:40.940008+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:27:40.940008+0000","last_peered":"2023-10-20T10:27:40.940008+0000","last_clean":"2023-10-20T10:27:40.940008+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:27:40.940008+0000","last_undegraded":"2023-10-20T10:27:40.940008+0000","last_fullsized":"2023-10-20T10:27:40.940008+0000","mapping_epoch":16,"log_start":"127'488631","ondisk_log_start":"127'488631","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3165,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 477s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7140765696,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":493063,"num_read_kb":1942404,"num_write":492747,"num_write_kb":1970988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":304,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:27:00.700075+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:27:00.700075+0000","last_peered":"2023-10-20T10:27:00.700075+0000","last_clean":"2023-10-20T10:27:00.700075+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:27:00.700075+0000","last_undegraded":"2023-10-20T10:27:00.700075+0000","last_fullsized":"2023-10-20T10:27:00.700075+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"129'463372","reported_seq":918136,"reported_epoch":129,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:27:41.346109+0000","last_change":"2023-10-20T10:19:59.154102+0000","last_active":"2023-10-20T10:27:41.346109+0000","last_peered":"2023-10-20T10:27:41.346109+0000","last_clean":"2023-10-20T10:27:41.346109+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:27:41.346109+0000","last_undegraded":"2023-10-20T10:27:41.346109+0000","last_fullsized":"2023-10-20T10:27:41.346109+0000","mapping_epoch":16,"log_start":"127'453342","ondisk_log_start":"127'453342","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3951,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 462s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6621040640,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":457515,"num_read_kb":1800192,"num_write":457055,"num_write_kb":1828220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6571,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:27:36.723255+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:27:36.723255+0000","last_peered":"2023-10-20T10:27:36.723255+0000","last_clean":"2023-10-20T10:27:36.723255+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:27:36.723255+0000","last_undegraded":"2023-10-20T10:27:36.723255+0000","last_fullsized":"2023-10-20T10:27:36.723255+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"129'491851","reported_seq":974757,"reported_epoch":129,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:27:40.933436+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:27:40.933436+0000","last_peered":"2023-10-20T10:27:40.933436+0000","last_clean":"2023-10-20T10:27:40.933436+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:27:40.933436+0000","last_undegraded":"2023-10-20T10:27:40.933436+0000","last_fullsized":"2023-10-20T10:27:40.933436+0000","mapping_epoch":16,"log_start":"127'481835","ondisk_log_start":"127'481835","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3467,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 475s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7033704448,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":485926,"num_read_kb":1912448,"num_write":485658,"num_write_kb":1942632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6418,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:27:36.758745+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:27:36.758745+0000","last_peered":"2023-10-20T10:27:36.758745+0000","last_clean":"2023-10-20T10:27:36.758745+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:27:36.758745+0000","last_undegraded":"2023-10-20T10:27:36.758745+0000","last_fullsized":"2023-10-20T10:27:36.758745+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"129'487227","reported_seq":968695,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:27:41.359046+0000","last_change":"2023-10-20T10:27:40.700969+0000","last_active":"2023-10-20T10:27:41.359046+0000","last_peered":"2023-10-20T10:27:41.359046+0000","last_clean":"2023-10-20T10:27:41.359046+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:27:41.359046+0000","last_undegraded":"2023-10-20T10:27:41.359046+0000","last_fullsized":"2023-10-20T10:27:41.359046+0000","mapping_epoch":16,"log_start":"127'477224","ondisk_log_start":"127'477224","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":6997962771,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":484574,"num_read_kb":1907392,"num_write":483753,"num_write_kb":1935006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":209,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:27:01.151359+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:27:01.151359+0000","last_peered":"2023-10-20T10:27:01.151359+0000","last_clean":"2023-10-20T10:27:01.151359+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:27:01.151359+0000","last_undegraded":"2023-10-20T10:27:01.151359+0000","last_fullsized":"2023-10-20T10:27:01.151359+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":302,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:26:13.156885+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:26:13.156885+0000","last_peered":"2023-10-20T10:26:13.156885+0000","last_clean":"2023-10-20T10:26:13.156885+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:26:13.156885+0000","last_undegraded":"2023-10-20T10:26:13.156885+0000","last_fullsized":"2023-10-20T10:26:13.156885+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27793473555,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1921078,"num_read_kb":7562436,"num_write":1919213,"num_write_kb":7676846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15721480192,"data_stored":29104234534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40106,"ondisk_log_size":40106,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":30236,"ondisk_log_size":30236,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739260,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9387468,"kb_used_data":7676228,"kb_used_omap":1745,"kb_used_meta":1709486,"kb_avail":84984372,"statfs":{"total":96636764160,"available":87023996928,"internally_reserved":0,"allocated":7860457472,"data_stored":14550965295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1787431,"internal_metadata":1750514137},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:27:08 2023","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.378,"15min":0.44500000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":0.55600000000000005,"5min":4.3799999999999999,"15min":8.2449999999999992},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.48699999999999999,"15min":0.48199999999999998},"min":{"1min":0.219,"5min":0.20799999999999999,"15min":0.18099999999999999},"max":{"1min":5.048,"5min":5.048,"15min":6.8239999999999998},"last":0.64200000000000002}]},{"osd":1,"last update":"Fri Oct 20 10:27:08 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":0.58099999999999996,"15min":0.70799999999999996},"min":{"1min":0.20999999999999999,"5min":0.19700000000000001,"15min":0.19500000000000001},"max":{"1min":6.1509999999999998,"5min":6.1509999999999998,"15min":35.670000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.46899999999999997,"15min":0.57299999999999995},"min":{"1min":0.23400000000000001,"5min":0.22800000000000001,"15min":0.214},"max":{"1min":4.8769999999999998,"5min":4.8769999999999998,"15min":34.948999999999998},"last":0.33100000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739260,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5450960,"kb_used_data":3764936,"kb_used_omap":676,"kb_used_meta":1685339,"kb_avail":88920880,"statfs":{"total":96636764160,"available":91054981120,"internally_reserved":0,"allocated":3855294464,"data_stored":7119464355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1725787723},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,5,1,16,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.50600000000000001,"15min":0.46800000000000003},"min":{"1min":0.20799999999999999,"5min":0.17899999999999999,"15min":0.157},"max":{"1min":4.9480000000000004,"5min":4.9480000000000004,"15min":4.9480000000000004},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.54800000000000004,"15min":0.498},"min":{"1min":0.20599999999999999,"5min":0.186,"15min":0.186},"max":{"1min":5.0049999999999999,"5min":5.4640000000000004,"15min":5.8479999999999999},"last":0.30099999999999999}]},{"osd":2,"last update":"Fri Oct 20 10:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.50700000000000001,"15min":0.41399999999999998},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":2.5659999999999998,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.55500000000000005,"15min":0.45900000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.23799999999999999},"max":{"1min":2.6019999999999999,"5min":4.8109999999999999,"15min":4.8109999999999999},"last":0.32400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739262,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5927268,"kb_used_data":3915228,"kb_used_omap":698,"kb_used_meta":2011333,"kb_avail":88444572,"statfs":{"total":96636764160,"available":90567241728,"internally_reserved":0,"allocated":4009193472,"data_stored":7434342676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":2059605578},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,3,10,6,8,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:26:52 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.72099999999999997,"15min":0.81499999999999995},"min":{"1min":0.19500000000000001,"5min":0.17799999999999999,"15min":0.16700000000000001},"max":{"1min":4.5609999999999999,"5min":8.0869999999999997,"15min":81.197000000000003},"last":6.3289999999999997},{"interface":"front","average":{"1min":0.28899999999999998,"5min":0.49399999999999999,"15min":0.72199999999999998},"min":{"1min":0.215,"5min":0.21299999999999999,"15min":0.184},"max":{"1min":0.52000000000000002,"5min":6.875,"15min":80.334999999999994},"last":6.2229999999999999}]},{"osd":2,"last update":"Fri Oct 20 10:26:52 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.371,"15min":0.48999999999999999},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.57199999999999995,"5min":2.3969999999999998,"15min":11.176},"last":6.2859999999999996},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.38400000000000001,"15min":0.52600000000000002},"min":{"1min":0.21099999999999999,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":0.752,"5min":1.093,"15min":11.093},"last":0.22800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4007555072,"data_stored":7433658368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3853549568,"data_stored":7118823443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7860375552,"data_stored":14551752723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:27:43.429 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:27:43.429 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:27:43.430 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:27:43.430 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:27:43.431 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:28:03.432 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:28:03.929 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:28:03.930 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:28:03.959 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2363,"stamp":"2023-10-20T10:28:03.293569+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27823010307,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1966848,"num_read_kb":7623501,"num_write":1927294,"num_write_kb":7707427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70646,"ondisk_log_size":70646,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20973236,"kb_used_data":15413080,"kb_used_omap":3120,"kb_used_meta":5557007,"kb_avail":262142284,"statfs":{"total":289910292480,"available":268433698816,"internally_reserved":0,"allocated":15782993920,"data_stored":29162820838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3195282,"internal_metadata":5690375790},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,11,6,9,20,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18059264,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4486,"num_read_kb":17904,"num_write":4411,"num_write_kb":17644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002051"},"pg_stats":[{"pgid":"2.7","version":"129'9790","reported_seq":14588,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:28:01.336164+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:28:01.336164+0000","last_peered":"2023-10-20T10:28:01.336164+0000","last_clean":"2023-10-20T10:28:01.336164+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:28:01.336164+0000","last_undegraded":"2023-10-20T10:28:01.336164+0000","last_fullsized":"2023-10-20T10:28:01.336164+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9790,"log_dups_size":0,"ondisk_log_size":9790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"129'19250","reported_seq":29828,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:28:01.351457+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:28:01.351457+0000","last_peered":"2023-10-20T10:28:01.351457+0000","last_clean":"2023-10-20T10:28:01.351457+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:28:01.351457+0000","last_undegraded":"2023-10-20T10:28:01.351457+0000","last_fullsized":"2023-10-20T10:28:01.351457+0000","mapping_epoch":13,"log_start":"104'9200","ondisk_log_start":"104'9200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":210,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:27:01.151399+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:27:01.151399+0000","last_peered":"2023-10-20T10:27:01.151399+0000","last_clean":"2023-10-20T10:27:01.151399+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:27:01.151399+0000","last_undegraded":"2023-10-20T10:27:01.151399+0000","last_fullsized":"2023-10-20T10:27:01.151399+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"129'9752","reported_seq":15325,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:28:01.334460+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:28:01.334460+0000","last_peered":"2023-10-20T10:28:01.334460+0000","last_clean":"2023-10-20T10:28:01.334460+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:28:01.334460+0000","last_undegraded":"2023-10-20T10:28:01.334460+0000","last_fullsized":"2023-10-20T10:28:01.334460+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9752,"log_dups_size":0,"ondisk_log_size":9752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"129'500579","reported_seq":992484,"reported_epoch":129,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:28:01.052701+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:28:01.052701+0000","last_peered":"2023-10-20T10:28:01.052701+0000","last_clean":"2023-10-20T10:28:01.052701+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:28:01.052701+0000","last_undegraded":"2023-10-20T10:28:01.052701+0000","last_fullsized":"2023-10-20T10:28:01.052701+0000","mapping_epoch":16,"log_start":"128'490531","ondisk_log_start":"128'490531","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3307,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 497s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7148138496,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":494905,"num_read_kb":1949748,"num_write":494530,"num_write_kb":1978120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":304,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:27:00.700075+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:27:00.700075+0000","last_peered":"2023-10-20T10:27:00.700075+0000","last_clean":"2023-10-20T10:27:00.700075+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:27:00.700075+0000","last_undegraded":"2023-10-20T10:27:00.700075+0000","last_fullsized":"2023-10-20T10:27:00.700075+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"129'465200","reported_seq":921657,"reported_epoch":129,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:28:01.378030+0000","last_change":"2023-10-20T10:19:59.154102+0000","last_active":"2023-10-20T10:28:01.378030+0000","last_peered":"2023-10-20T10:28:01.378030+0000","last_clean":"2023-10-20T10:28:01.378030+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:28:01.378030+0000","last_undegraded":"2023-10-20T10:28:01.378030+0000","last_fullsized":"2023-10-20T10:28:01.378030+0000","mapping_epoch":16,"log_start":"127'455142","ondisk_log_start":"127'455142","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":4113,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 482s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6628012032,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":459197,"num_read_kb":1806916,"num_write":458760,"num_write_kb":1835040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6587,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:27:57.203999+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:27:57.203999+0000","last_peered":"2023-10-20T10:27:57.203999+0000","last_clean":"2023-10-20T10:27:57.203999+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:27:57.203999+0000","last_undegraded":"2023-10-20T10:27:57.203999+0000","last_fullsized":"2023-10-20T10:27:57.203999+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"129'493755","reported_seq":978420,"reported_epoch":129,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:28:01.050086+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:28:01.050086+0000","last_peered":"2023-10-20T10:28:01.050086+0000","last_clean":"2023-10-20T10:28:01.050086+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:28:01.050086+0000","last_undegraded":"2023-10-20T10:28:01.050086+0000","last_fullsized":"2023-10-20T10:28:01.050086+0000","mapping_epoch":16,"log_start":"128'483736","ondisk_log_start":"128'483736","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3616,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 496s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7041028096,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":487675,"num_read_kb":1919432,"num_write":487450,"num_write_kb":1949800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6434,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:27:57.207947+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:27:57.207947+0000","last_peered":"2023-10-20T10:27:57.207947+0000","last_clean":"2023-10-20T10:27:57.207947+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:27:57.207947+0000","last_undegraded":"2023-10-20T10:27:57.207947+0000","last_fullsized":"2023-10-20T10:27:57.207947+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"129'489014","reported_seq":972316,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:28:01.382300+0000","last_change":"2023-10-20T10:27:56.837371+0000","last_active":"2023-10-20T10:28:01.382300+0000","last_peered":"2023-10-20T10:28:01.382300+0000","last_clean":"2023-10-20T10:28:01.382300+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:28:01.382300+0000","last_undegraded":"2023-10-20T10:28:01.382300+0000","last_fullsized":"2023-10-20T10:28:01.382300+0000","mapping_epoch":16,"log_start":"127'478924","ondisk_log_start":"127'478924","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7005237267,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":486400,"num_read_kb":1914684,"num_write":485540,"num_write_kb":1942154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":209,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:27:01.151359+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:27:01.151359+0000","last_peered":"2023-10-20T10:27:01.151359+0000","last_clean":"2023-10-20T10:27:01.151359+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:27:01.151359+0000","last_undegraded":"2023-10-20T10:27:01.151359+0000","last_fullsized":"2023-10-20T10:27:01.151359+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":303,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:27:23.156228+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:27:23.156228+0000","last_peered":"2023-10-20T10:27:23.156228+0000","last_clean":"2023-10-20T10:27:23.156228+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:27:23.156228+0000","last_undegraded":"2023-10-20T10:27:23.156228+0000","last_fullsized":"2023-10-20T10:27:23.156228+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27822415891,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1928177,"num_read_kb":7590780,"num_write":1926280,"num_write_kb":7705114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15779450880,"data_stored":29162205222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40215,"ondisk_log_size":40215,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":30332,"ondisk_log_size":30332,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739264,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9494508,"kb_used_data":7704608,"kb_used_omap":1745,"kb_used_meta":1788142,"kb_avail":84877332,"statfs":{"total":96636764160,"available":86914387968,"internally_reserved":0,"allocated":7889518592,"data_stored":14580026415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1787431,"internal_metadata":1831057881},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:27:08 2023","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.378,"15min":0.44500000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":0.55600000000000005,"5min":4.3799999999999999,"15min":8.2449999999999992},"last":0.20100000000000001},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.48699999999999999,"15min":0.48199999999999998},"min":{"1min":0.219,"5min":0.20799999999999999,"15min":0.18099999999999999},"max":{"1min":5.048,"5min":5.048,"15min":6.8239999999999998},"last":0.33100000000000002}]},{"osd":1,"last update":"Fri Oct 20 10:27:08 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":0.58099999999999996,"15min":0.70799999999999996},"min":{"1min":0.20999999999999999,"5min":0.19700000000000001,"15min":0.19500000000000001},"max":{"1min":6.1509999999999998,"5min":6.1509999999999998,"15min":35.670000000000002},"last":0.371},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.46899999999999997,"15min":0.57299999999999995},"min":{"1min":0.23400000000000001,"5min":0.22800000000000001,"15min":0.214},"max":{"1min":4.8769999999999998,"5min":4.8769999999999998,"15min":34.948999999999998},"last":0.29499999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739264,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5501100,"kb_used_data":3778980,"kb_used_omap":676,"kb_used_meta":1721435,"kb_avail":88870740,"statfs":{"total":96636764160,"available":91003637760,"internally_reserved":0,"allocated":3869675520,"data_stored":7133845411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1762750027},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,3,3,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:28:00 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.48699999999999999,"15min":0.47599999999999998},"min":{"1min":0.20599999999999999,"5min":0.188,"15min":0.17199999999999999},"max":{"1min":3,"5min":4.9480000000000004,"15min":4.9480000000000004},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.52900000000000003,"15min":0.48299999999999998},"min":{"1min":0.19400000000000001,"5min":0.186,"15min":0.186},"max":{"1min":3.0779999999999998,"5min":5.4640000000000004,"15min":5.8479999999999999},"last":0.216}]},{"osd":2,"last update":"Fri Oct 20 10:28:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.53700000000000003,"15min":0.435},"min":{"1min":0.23000000000000001,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":7.6459999999999999,"5min":7.6459999999999999,"15min":7.6459999999999999},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.58199999999999996,"15min":0.48099999999999998},"min":{"1min":0.27900000000000003,"5min":0.253,"15min":0.23799999999999999},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.35399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739266,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5977628,"kb_used_data":3929492,"kb_used_omap":698,"kb_used_meta":2047429,"kb_avail":88394212,"statfs":{"total":96636764160,"available":90515673088,"internally_reserved":0,"allocated":4023799808,"data_stored":7448949012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":2096567882},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,2,14],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:27:53 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.80000000000000004,"15min":0.81599999999999995},"min":{"1min":0.20100000000000001,"5min":0.17899999999999999,"15min":0.16700000000000001},"max":{"1min":6.3289999999999997,"5min":8.0869999999999997,"15min":81.197000000000003},"last":0.313},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.58899999999999997,"15min":0.752},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.184},"max":{"1min":6.2229999999999999,"5min":6.875,"15min":80.334999999999994},"last":0.28100000000000003}]},{"osd":2,"last update":"Fri Oct 20 10:27:53 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.48399999999999999,"15min":0.51000000000000001},"min":{"1min":0.24299999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.2859999999999996,"5min":6.2859999999999996,"15min":11.176},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.44500000000000001,"15min":0.52500000000000002},"min":{"1min":0.22800000000000001,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":4.4260000000000002,"5min":4.4260000000000002,"15min":11.093},"last":0.45500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4022198272,"data_stored":7448301568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3867848704,"data_stored":7133122579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7889403904,"data_stored":14580781075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:28:03.961 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:28:03.962 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:28:03.962 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:28:03.962 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:28:03.963 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:28:23.965 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:28:24.419 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:28:24.420 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:28:24.497 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2373,"stamp":"2023-10-20T10:28:23.297206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27851289091,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1973880,"num_read_kb":7651533,"num_write":1934206,"num_write_kb":7735075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70687,"ondisk_log_size":70687,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21182160,"kb_used_data":15468088,"kb_used_omap":3120,"kb_used_meta":5710927,"kb_avail":261933360,"statfs":{"total":289910292480,"available":268219760640,"internally_reserved":0,"allocated":15839322112,"data_stored":29219149030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3195282,"internal_metadata":5847989870},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,7,10,7,10,18,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17727488,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4311,"num_read_kb":17188,"num_write":4331,"num_write_kb":17324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002100"},"pg_stats":[{"pgid":"2.7","version":"129'9820","reported_seq":14630,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:28:20.597532+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:28:20.597532+0000","last_peered":"2023-10-20T10:28:20.597532+0000","last_clean":"2023-10-20T10:28:20.597532+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:28:20.597532+0000","last_undegraded":"2023-10-20T10:28:20.597532+0000","last_fullsized":"2023-10-20T10:28:20.597532+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9820,"log_dups_size":0,"ondisk_log_size":9820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"129'19310","reported_seq":29912,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:28:20.614352+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:28:20.614352+0000","last_peered":"2023-10-20T10:28:20.614352+0000","last_clean":"2023-10-20T10:28:20.614352+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:28:20.614352+0000","last_undegraded":"2023-10-20T10:28:20.614352+0000","last_fullsized":"2023-10-20T10:28:20.614352+0000","mapping_epoch":13,"log_start":"105'9300","ondisk_log_start":"105'9300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":210,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:27:01.151399+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:27:01.151399+0000","last_peered":"2023-10-20T10:27:01.151399+0000","last_clean":"2023-10-20T10:27:01.151399+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:27:01.151399+0000","last_undegraded":"2023-10-20T10:27:01.151399+0000","last_fullsized":"2023-10-20T10:27:01.151399+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"129'9782","reported_seq":15367,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:28:20.584318+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:28:20.584318+0000","last_peered":"2023-10-20T10:28:20.584318+0000","last_clean":"2023-10-20T10:28:20.584318+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:28:20.584318+0000","last_undegraded":"2023-10-20T10:28:20.584318+0000","last_fullsized":"2023-10-20T10:28:20.584318+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9782,"log_dups_size":0,"ondisk_log_size":9782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"129'502499","reported_seq":996187,"reported_epoch":129,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:28:21.249142+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:28:21.249142+0000","last_peered":"2023-10-20T10:28:21.249142+0000","last_clean":"2023-10-20T10:28:21.249142+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:28:21.249142+0000","last_undegraded":"2023-10-20T10:28:21.249142+0000","last_fullsized":"2023-10-20T10:28:21.249142+0000","mapping_epoch":16,"log_start":"128'492431","ondisk_log_start":"128'492431","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3459,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 517s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7155519488,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":496678,"num_read_kb":1956812,"num_write":496337,"num_write_kb":1985348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":304,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:27:00.700075+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:27:00.700075+0000","last_peered":"2023-10-20T10:27:00.700075+0000","last_clean":"2023-10-20T10:27:00.700075+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:27:00.700075+0000","last_undegraded":"2023-10-20T10:27:00.700075+0000","last_fullsized":"2023-10-20T10:27:00.700075+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"129'466983","reported_seq":925136,"reported_epoch":129,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:28:21.478400+0000","last_change":"2023-10-20T10:19:59.154102+0000","last_active":"2023-10-20T10:28:21.478400+0000","last_peered":"2023-10-20T10:28:21.478400+0000","last_clean":"2023-10-20T10:28:21.478400+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:28:21.478400+0000","last_undegraded":"2023-10-20T10:28:21.478400+0000","last_fullsized":"2023-10-20T10:28:21.478400+0000","mapping_epoch":16,"log_start":"128'456942","ondisk_log_start":"128'456942","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":4315,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 502s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6634684416,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":460876,"num_read_kb":1813612,"num_write":460389,"num_write_kb":1841556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6603,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:28:17.678723+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:28:17.678723+0000","last_peered":"2023-10-20T10:28:17.678723+0000","last_clean":"2023-10-20T10:28:17.678723+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:28:17.678723+0000","last_undegraded":"2023-10-20T10:28:17.678723+0000","last_fullsized":"2023-10-20T10:28:17.678723+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"129'495635","reported_seq":982103,"reported_epoch":129,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:28:21.216819+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:28:21.216819+0000","last_peered":"2023-10-20T10:28:21.216819+0000","last_clean":"2023-10-20T10:28:21.216819+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:28:21.216819+0000","last_undegraded":"2023-10-20T10:28:21.216819+0000","last_fullsized":"2023-10-20T10:28:21.216819+0000","mapping_epoch":16,"log_start":"128'485536","ondisk_log_start":"128'485536","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3800,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 516s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7048142848,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":489465,"num_read_kb":1926556,"num_write":489188,"num_write_kb":1956752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6451,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:28:17.681320+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:28:17.681320+0000","last_peered":"2023-10-20T10:28:17.681320+0000","last_clean":"2023-10-20T10:28:17.681320+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:28:17.681320+0000","last_undegraded":"2023-10-20T10:28:17.681320+0000","last_fullsized":"2023-10-20T10:28:17.681320+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"129'490752","reported_seq":975849,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:28:21.484395+0000","last_change":"2023-10-20T10:28:20.735659+0000","last_active":"2023-10-20T10:28:21.484395+0000","last_peered":"2023-10-20T10:28:21.484395+0000","last_clean":"2023-10-20T10:28:21.484395+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:28:21.484395+0000","last_undegraded":"2023-10-20T10:28:21.484395+0000","last_fullsized":"2023-10-20T10:28:21.484395+0000","mapping_epoch":16,"log_start":"128'480724","ondisk_log_start":"128'480724","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7012347923,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":488190,"num_read_kb":1921832,"num_write":487278,"num_write_kb":1949106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":209,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-20T10:27:01.151359+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:27:01.151359+0000","last_peered":"2023-10-20T10:27:01.151359+0000","last_clean":"2023-10-20T10:27:01.151359+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:27:01.151359+0000","last_undegraded":"2023-10-20T10:27:01.151359+0000","last_fullsized":"2023-10-20T10:27:01.151359+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":303,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:27:23.156228+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:27:23.156228+0000","last_peered":"2023-10-20T10:27:23.156228+0000","last_clean":"2023-10-20T10:27:23.156228+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:27:23.156228+0000","last_undegraded":"2023-10-20T10:27:23.156228+0000","last_fullsized":"2023-10-20T10:27:23.156228+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27850694675,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1935209,"num_read_kb":7618812,"num_write":1933192,"num_write_kb":7732762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15835885568,"data_stored":29218639910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40236,"ondisk_log_size":40236,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":30352,"ondisk_log_size":30352,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739268,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9577584,"kb_used_data":7732264,"kb_used_omap":1745,"kb_used_meta":1843566,"kb_avail":84794256,"statfs":{"total":96636764160,"available":86829318144,"internally_reserved":0,"allocated":7917838336,"data_stored":14608346159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1787431,"internal_metadata":1887812057},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.39800000000000002,"15min":0.45600000000000002},"min":{"1min":0.189,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":2.8919999999999999,"5min":4.3799999999999999,"15min":8.2449999999999992},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.51100000000000001,"15min":0.48899999999999999},"min":{"1min":0.22,"5min":0.20799999999999999,"15min":0.18099999999999999},"max":{"1min":2.855,"5min":5.048,"15min":6.8239999999999998},"last":0.33200000000000002}]},{"osd":1,"last update":"Fri Oct 20 10:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.54400000000000004,"15min":0.69999999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":1.8380000000000001,"5min":6.1509999999999998,"15min":35.670000000000002},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.46200000000000002,"15min":0.56999999999999995},"min":{"1min":0.23799999999999999,"5min":0.22900000000000001,"15min":0.214},"max":{"1min":1.865,"5min":4.8769999999999998,"15min":34.948999999999998},"last":0.27800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739268,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5562924,"kb_used_data":3792228,"kb_used_omap":676,"kb_used_meta":1770011,"kb_avail":88808916,"statfs":{"total":96636764160,"available":90940329984,"internally_reserved":0,"allocated":3883241472,"data_stored":7147411363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1812491851},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,2,1,4,13,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:28:00 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.48699999999999999,"15min":0.47599999999999998},"min":{"1min":0.20599999999999999,"5min":0.188,"15min":0.17199999999999999},"max":{"1min":3,"5min":4.9480000000000004,"15min":4.9480000000000004},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.52900000000000003,"15min":0.48299999999999998},"min":{"1min":0.19400000000000001,"5min":0.186,"15min":0.186},"max":{"1min":3.0779999999999998,"5min":5.4640000000000004,"15min":5.8479999999999999},"last":0.26900000000000002}]},{"osd":2,"last update":"Fri Oct 20 10:28:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.53700000000000003,"15min":0.435},"min":{"1min":0.23000000000000001,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":7.6459999999999999,"5min":7.6459999999999999,"15min":7.6459999999999999},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.58199999999999996,"15min":0.48099999999999998},"min":{"1min":0.27900000000000003,"5min":0.253,"15min":0.23799999999999999},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.313}]}]},{"osd":0,"up_from":8,"seq":34359739270,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6041652,"kb_used_data":3943596,"kb_used_omap":698,"kb_used_meta":2097349,"kb_avail":88330188,"statfs":{"total":96636764160,"available":90450112512,"internally_reserved":0,"allocated":4038242304,"data_stored":7463391508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":715190,"internal_metadata":2147685962},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,6,2,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:27:53 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.80000000000000004,"15min":0.81599999999999995},"min":{"1min":0.20100000000000001,"5min":0.17899999999999999,"15min":0.16700000000000001},"max":{"1min":6.3289999999999997,"5min":8.0869999999999997,"15min":81.197000000000003},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.58899999999999997,"15min":0.752},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.184},"max":{"1min":6.2229999999999999,"5min":6.875,"15min":80.334999999999994},"last":0.25800000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:27:53 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.48399999999999999,"15min":0.51000000000000001},"min":{"1min":0.24299999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.2859999999999996,"5min":6.2859999999999996,"15min":11.176},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.44500000000000001,"15min":0.52500000000000002},"min":{"1min":0.22800000000000001,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":4.4260000000000002,"5min":4.4260000000000002,"15min":11.093},"last":0.317}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4036714496,"data_stored":7462817792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3881644032,"data_stored":7146917907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7917527040,"data_stored":14608904211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:28:24.499 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:28:24.499 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:28:24.500 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:28:24.500 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:28:24.501 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-20T10:28:24.993 INFO:teuthology.orchestra.run.smithi150.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-20T10:28:25.029 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-20T10:28:25.543 INFO:teuthology.orchestra.run.smithi150.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-20T10:28:25.621 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-20T10:28:26.115 INFO:teuthology.orchestra.run.smithi150.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-20T10:28:26.151 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-20T10:28:26.661 INFO:teuthology.orchestra.run.smithi150.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-20T10:28:26.704 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:28:44.267 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:28:44.267+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:28:44.268 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:28:44.267+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:28:46.705 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:28:47.229 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:28:47.229 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:28:47.259 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2385,"stamp":"2023-10-20T10:28:45.300977+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27879854595,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1980887,"num_read_kb":7679441,"num_write":1941191,"num_write_kb":7763015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70750,"ondisk_log_size":70750,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20807252,"kb_used_data":15524540,"kb_used_omap":3122,"kb_used_meta":5279565,"kb_avail":262308268,"statfs":{"total":289910292480,"available":268603666432,"internally_reserved":0,"allocated":15897128960,"data_stored":29276932322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3196961,"internal_metadata":5406275551},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,3,3,6,11,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14520320,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3489,"num_read_kb":13900,"num_write":3546,"num_write_kb":14184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002155"},"pg_stats":[{"pgid":"2.7","version":"130'9847","reported_seq":14670,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:28:40.914030+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:28:40.914030+0000","last_peered":"2023-10-20T10:28:40.914030+0000","last_clean":"2023-10-20T10:28:40.914030+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:28:40.914030+0000","last_undegraded":"2023-10-20T10:28:40.914030+0000","last_fullsized":"2023-10-20T10:28:40.914030+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9847,"log_dups_size":0,"ondisk_log_size":9847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"130'19364","reported_seq":29991,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:28:40.951938+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:28:40.951938+0000","last_peered":"2023-10-20T10:28:40.951938+0000","last_clean":"2023-10-20T10:28:40.951938+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:28:40.951938+0000","last_undegraded":"2023-10-20T10:28:40.951938+0000","last_fullsized":"2023-10-20T10:28:40.951938+0000","mapping_epoch":13,"log_start":"105'9300","ondisk_log_start":"105'9300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":211,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:27:23.149676+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:27:23.149676+0000","last_peered":"2023-10-20T10:27:23.149676+0000","last_clean":"2023-10-20T10:27:23.149676+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:27:23.149676+0000","last_undegraded":"2023-10-20T10:27:23.149676+0000","last_fullsized":"2023-10-20T10:27:23.149676+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"130'9809","reported_seq":15407,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:28:40.903926+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:28:40.903926+0000","last_peered":"2023-10-20T10:28:40.903926+0000","last_clean":"2023-10-20T10:28:40.903926+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:28:40.903926+0000","last_undegraded":"2023-10-20T10:28:40.903926+0000","last_fullsized":"2023-10-20T10:28:40.903926+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9809,"log_dups_size":0,"ondisk_log_size":9809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"130'504412","reported_seq":999894,"reported_epoch":130,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:28:41.418496+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:28:41.418496+0000","last_peered":"2023-10-20T10:28:41.418496+0000","last_clean":"2023-10-20T10:28:41.418496+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:28:41.418496+0000","last_undegraded":"2023-10-20T10:28:41.418496+0000","last_fullsized":"2023-10-20T10:28:41.418496+0000","mapping_epoch":16,"log_start":"128'494332","ondisk_log_start":"128'494332","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3604,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 537s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7162900480,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":498460,"num_read_kb":1963912,"num_write":498137,"num_write_kb":1992548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":305,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:27:23.166954+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:27:23.166954+0000","last_peered":"2023-10-20T10:27:23.166954+0000","last_clean":"2023-10-20T10:27:23.166954+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:27:23.166954+0000","last_undegraded":"2023-10-20T10:27:23.166954+0000","last_fullsized":"2023-10-20T10:27:23.166954+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"130'468827","reported_seq":928684,"reported_epoch":130,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:28:41.701324+0000","last_change":"2023-10-20T10:19:59.154102+0000","last_active":"2023-10-20T10:28:41.701324+0000","last_peered":"2023-10-20T10:28:41.701324+0000","last_clean":"2023-10-20T10:28:41.701324+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:28:41.701324+0000","last_undegraded":"2023-10-20T10:28:41.701324+0000","last_fullsized":"2023-10-20T10:28:41.701324+0000","mapping_epoch":16,"log_start":"128'458742","ondisk_log_start":"128'458742","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":4499,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 522s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6641635328,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":462564,"num_read_kb":1820336,"num_write":462092,"num_write_kb":1848368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6620,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:28:38.165816+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:28:38.165816+0000","last_peered":"2023-10-20T10:28:38.165816+0000","last_clean":"2023-10-20T10:28:38.165816+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:28:38.165816+0000","last_undegraded":"2023-10-20T10:28:38.165816+0000","last_fullsized":"2023-10-20T10:28:38.165816+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"130'497548","reported_seq":985769,"reported_epoch":130,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:28:41.390355+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:28:41.390355+0000","last_peered":"2023-10-20T10:28:41.390355+0000","last_clean":"2023-10-20T10:28:41.390355+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:28:41.390355+0000","last_undegraded":"2023-10-20T10:28:41.390355+0000","last_fullsized":"2023-10-20T10:28:41.390355+0000","mapping_epoch":16,"log_start":"128'487536","ondisk_log_start":"128'487536","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3960,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 536s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7055491072,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":491198,"num_read_kb":1933468,"num_write":490984,"num_write_kb":1963936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6468,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:28:38.176732+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:28:38.176732+0000","last_peered":"2023-10-20T10:28:38.176732+0000","last_clean":"2023-10-20T10:28:38.176732+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:28:38.176732+0000","last_undegraded":"2023-10-20T10:28:38.176732+0000","last_fullsized":"2023-10-20T10:28:38.176732+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"130'492438","reported_seq":979347,"reported_epoch":130,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:28:41.707451+0000","last_change":"2023-10-20T10:28:41.683965+0000","last_active":"2023-10-20T10:28:41.707451+0000","last_peered":"2023-10-20T10:28:41.707451+0000","last_clean":"2023-10-20T10:28:41.707451+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:28:41.707451+0000","last_undegraded":"2023-10-20T10:28:41.707451+0000","last_fullsized":"2023-10-20T10:28:41.707451+0000","mapping_epoch":16,"log_start":"128'482424","ondisk_log_start":"128'482424","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7019233299,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":489994,"num_read_kb":1929004,"num_write":488964,"num_write_kb":1955850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":210,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:27:23.149665+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:27:23.149665+0000","last_peered":"2023-10-20T10:27:23.149665+0000","last_clean":"2023-10-20T10:27:23.149665+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:27:23.149665+0000","last_undegraded":"2023-10-20T10:27:23.149665+0000","last_fullsized":"2023-10-20T10:27:23.149665+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":304,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:28:01.391495+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:28:01.391495+0000","last_peered":"2023-10-20T10:28:01.391495+0000","last_clean":"2023-10-20T10:28:01.391495+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:28:01.391495+0000","last_undegraded":"2023-10-20T10:28:01.391495+0000","last_fullsized":"2023-10-20T10:28:01.391495+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27879260179,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1942216,"num_read_kb":7646720,"num_write":1940177,"num_write_kb":7760702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15893008384,"data_stored":29275762726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40191,"ondisk_log_size":40191,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":30460,"ondisk_log_size":30460,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739272,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9708312,"kb_used_data":7760272,"kb_used_omap":1745,"kb_used_meta":1946286,"kb_avail":84663528,"statfs":{"total":96636764160,"available":86695452672,"internally_reserved":0,"allocated":7946518528,"data_stored":14637018499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1787431,"internal_metadata":1992997337},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.39800000000000002,"15min":0.45600000000000002},"min":{"1min":0.189,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":2.8919999999999999,"5min":4.3799999999999999,"15min":8.2449999999999992},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.51100000000000001,"15min":0.48899999999999999},"min":{"1min":0.22,"5min":0.20799999999999999,"15min":0.18099999999999999},"max":{"1min":2.855,"5min":5.048,"15min":6.8239999999999998},"last":0.33400000000000002}]},{"osd":1,"last update":"Fri Oct 20 10:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.54400000000000004,"15min":0.69999999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":1.8380000000000001,"5min":6.1509999999999998,"15min":35.670000000000002},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.46200000000000002,"15min":0.56999999999999995},"min":{"1min":0.23799999999999999,"5min":0.22900000000000001,"15min":0.214},"max":{"1min":1.865,"5min":4.8769999999999998,"15min":34.948999999999998},"last":0.38200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739272,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5631224,"kb_used_data":3806384,"kb_used_omap":676,"kb_used_meta":1824155,"kb_avail":88740616,"statfs":{"total":96636764160,"available":90870390784,"internally_reserved":0,"allocated":3897737216,"data_stored":7161899255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1867935307},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:28:00 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.48699999999999999,"15min":0.47599999999999998},"min":{"1min":0.20599999999999999,"5min":0.188,"15min":0.17199999999999999},"max":{"1min":3,"5min":4.9480000000000004,"15min":4.9480000000000004},"last":2.7839999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.52900000000000003,"15min":0.48299999999999998},"min":{"1min":0.19400000000000001,"5min":0.186,"15min":0.186},"max":{"1min":3.0779999999999998,"5min":5.4640000000000004,"15min":5.8479999999999999},"last":0.218}]},{"osd":2,"last update":"Fri Oct 20 10:28:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.53700000000000003,"15min":0.435},"min":{"1min":0.23000000000000001,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":7.6459999999999999,"5min":7.6459999999999999,"15min":7.6459999999999999},"last":2.8199999999999998},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.58199999999999996,"15min":0.48099999999999998},"min":{"1min":0.27900000000000003,"5min":0.253,"15min":0.23799999999999999},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":2.8919999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739274,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5467716,"kb_used_data":3957884,"kb_used_omap":700,"kb_used_meta":1509123,"kb_avail":88904124,"statfs":{"total":96636764160,"available":91037822976,"internally_reserved":0,"allocated":4052873216,"data_stored":7478014568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":716869,"internal_metadata":1545342907},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,0,2,8,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:27:53 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.80000000000000004,"15min":0.81599999999999995},"min":{"1min":0.20100000000000001,"5min":0.17899999999999999,"15min":0.16700000000000001},"max":{"1min":6.3289999999999997,"5min":8.0869999999999997,"15min":81.197000000000003},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.58899999999999997,"15min":0.752},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.184},"max":{"1min":6.2229999999999999,"5min":6.875,"15min":80.334999999999994},"last":0.43099999999999999}]},{"osd":2,"last update":"Fri Oct 20 10:27:53 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.48399999999999999,"15min":0.51000000000000001},"min":{"1min":0.24299999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.2859999999999996,"5min":6.2859999999999996,"15min":11.176},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.44500000000000001,"15min":0.52500000000000002},"min":{"1min":0.22800000000000001,"5min":0.20599999999999999,"15min":0.187},"max":{"1min":4.4260000000000002,"5min":4.4260000000000002,"15min":11.093},"last":0.50900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4051451904,"data_stored":7477555200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3895533568,"data_stored":7160807443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7946022912,"data_stored":14637400083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:28:47.261 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:28:47.261 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:28:47.262 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:28:47.262 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:28:47.262 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:29:07.263 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:29:07.271 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:29:07.266+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:29:07.300 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:29:07.298+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:29:07.839 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:29:07.840 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:29:07.872 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2396,"stamp":"2023-10-20T10:29:07.304728+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27916329475,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1989712,"num_read_kb":7714537,"num_write":1950142,"num_write_kb":7798819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70763,"ondisk_log_size":70763,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21105584,"kb_used_data":15595544,"kb_used_omap":3122,"kb_used_meta":5506893,"kb_avail":262009936,"statfs":{"total":289910292480,"available":268298174464,"internally_reserved":0,"allocated":15969837056,"data_stored":29349640418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3196961,"internal_metadata":5639059423},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2,11,18,20,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21749760,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5223,"num_read_kb":20780,"num_write":5339,"num_write_kb":21356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001986"},"pg_stats":[{"pgid":"2.7","version":"130'9878","reported_seq":14716,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:29:06.681749+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:29:06.681749+0000","last_peered":"2023-10-20T10:29:06.681749+0000","last_clean":"2023-10-20T10:29:06.681749+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:29:06.681749+0000","last_undegraded":"2023-10-20T10:29:06.681749+0000","last_fullsized":"2023-10-20T10:29:06.681749+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9878,"log_dups_size":0,"ondisk_log_size":9878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"130'19424","reported_seq":30081,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:29:06.724187+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:29:06.724187+0000","last_peered":"2023-10-20T10:29:06.724187+0000","last_clean":"2023-10-20T10:29:06.724187+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:29:06.724187+0000","last_undegraded":"2023-10-20T10:29:06.724187+0000","last_fullsized":"2023-10-20T10:29:06.724187+0000","mapping_epoch":13,"log_start":"105'9400","ondisk_log_start":"105'9400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":212,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:28:28.174975+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:28:28.174975+0000","last_peered":"2023-10-20T10:28:28.174975+0000","last_clean":"2023-10-20T10:28:28.174975+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:28:28.174975+0000","last_undegraded":"2023-10-20T10:28:28.174975+0000","last_fullsized":"2023-10-20T10:28:28.174975+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"130'9840","reported_seq":15454,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:29:06.691291+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:29:06.691291+0000","last_peered":"2023-10-20T10:29:06.691291+0000","last_clean":"2023-10-20T10:29:06.691291+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:29:06.691291+0000","last_undegraded":"2023-10-20T10:29:06.691291+0000","last_fullsized":"2023-10-20T10:29:06.691291+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9840,"log_dups_size":0,"ondisk_log_size":9840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"130'506851","reported_seq":1004610,"reported_epoch":130,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:29:06.752641+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:29:06.752641+0000","last_peered":"2023-10-20T10:29:06.752641+0000","last_clean":"2023-10-20T10:29:06.752641+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:29:06.752641+0000","last_undegraded":"2023-10-20T10:29:06.752641+0000","last_fullsized":"2023-10-20T10:29:06.752641+0000","mapping_epoch":16,"log_start":"128'496832","ondisk_log_start":"128'496832","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3777,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 562s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7172304896,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":500717,"num_read_kb":1972876,"num_write":500447,"num_write_kb":2001788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":306,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:28:28.161666+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:28:28.161666+0000","last_peered":"2023-10-20T10:28:28.161666+0000","last_clean":"2023-10-20T10:28:28.161666+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:28:28.161666+0000","last_undegraded":"2023-10-20T10:28:28.161666+0000","last_fullsized":"2023-10-20T10:28:28.161666+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"130'470983","reported_seq":932898,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:29:06.821231+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:29:06.821231+0000","last_peered":"2023-10-20T10:29:06.821231+0000","last_clean":"2023-10-20T10:29:06.821231+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:29:06.821231+0000","last_undegraded":"2023-10-20T10:29:06.821231+0000","last_fullsized":"2023-10-20T10:29:06.821231+0000","mapping_epoch":16,"log_start":"128'460942","ondisk_log_start":"128'460942","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6650392576,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":464617,"num_read_kb":1828496,"num_write":464239,"num_write_kb":1856956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6641,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:29:03.771889+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:29:03.771889+0000","last_peered":"2023-10-20T10:29:03.771889+0000","last_clean":"2023-10-20T10:29:03.771889+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:29:03.771889+0000","last_undegraded":"2023-10-20T10:29:03.771889+0000","last_fullsized":"2023-10-20T10:29:03.771889+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"130'499982","reported_seq":990481,"reported_epoch":130,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:29:06.744079+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:29:06.744079+0000","last_peered":"2023-10-20T10:29:06.744079+0000","last_clean":"2023-10-20T10:29:06.744079+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:29:06.744079+0000","last_undegraded":"2023-10-20T10:29:06.744079+0000","last_fullsized":"2023-10-20T10:29:06.744079+0000","mapping_epoch":16,"log_start":"128'489936","ondisk_log_start":"128'489936","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":4208,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 561s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7064641536,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":493462,"num_read_kb":1942484,"num_write":493231,"num_write_kb":1972924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6488,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:29:03.758780+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:29:03.758780+0000","last_peered":"2023-10-20T10:29:03.758780+0000","last_clean":"2023-10-20T10:29:03.758780+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:29:03.758780+0000","last_undegraded":"2023-10-20T10:29:03.758780+0000","last_fullsized":"2023-10-20T10:29:03.758780+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"130'494800","reported_seq":983980,"reported_epoch":130,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:29:06.814632+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:29:06.814632+0000","last_peered":"2023-10-20T10:29:06.814632+0000","last_clean":"2023-10-20T10:29:06.814632+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:29:06.814632+0000","last_undegraded":"2023-10-20T10:29:06.814632+0000","last_fullsized":"2023-10-20T10:29:06.814632+0000","mapping_epoch":16,"log_start":"128'484724","ondisk_log_start":"128'484724","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":151,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 22s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7028396051,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":492245,"num_read_kb":1937960,"num_write":491211,"num_write_kb":1964838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":211,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:28:28.174929+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:28:28.174929+0000","last_peered":"2023-10-20T10:28:28.174929+0000","last_clean":"2023-10-20T10:28:28.174929+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:28:28.174929+0000","last_undegraded":"2023-10-20T10:28:28.174929+0000","last_fullsized":"2023-10-20T10:28:28.174929+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":304,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:28:01.391495+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:28:01.391495+0000","last_peered":"2023-10-20T10:28:01.391495+0000","last_clean":"2023-10-20T10:28:01.391495+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:28:01.391495+0000","last_undegraded":"2023-10-20T10:28:01.391495+0000","last_fullsized":"2023-10-20T10:28:01.391495+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27915735059,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1951041,"num_read_kb":7681816,"num_write":1949128,"num_write_kb":7796506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15966158848,"data_stored":29348913190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40182,"ondisk_log_size":40182,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":30482,"ondisk_log_size":30482,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739277,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9845604,"kb_used_data":7795804,"kb_used_omap":1745,"kb_used_meta":2048046,"kb_avail":84526236,"statfs":{"total":96636764160,"available":86554865664,"internally_reserved":0,"allocated":7982903296,"data_stored":14673403267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1787431,"internal_metadata":2097199577},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.39800000000000002,"15min":0.45600000000000002},"min":{"1min":0.189,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":2.8919999999999999,"5min":4.3799999999999999,"15min":8.2449999999999992},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.51100000000000001,"15min":0.48899999999999999},"min":{"1min":0.22,"5min":0.20799999999999999,"15min":0.18099999999999999},"max":{"1min":2.855,"5min":5.048,"15min":6.8239999999999998},"last":0.27800000000000002}]},{"osd":1,"last update":"Fri Oct 20 10:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.54400000000000004,"15min":0.69999999999999996},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":1.8380000000000001,"5min":6.1509999999999998,"15min":35.670000000000002},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.46200000000000002,"15min":0.56999999999999995},"min":{"1min":0.23799999999999999,"5min":0.22900000000000001,"15min":0.214},"max":{"1min":1.865,"5min":4.8769999999999998,"15min":34.948999999999998},"last":0.40300000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739277,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5690548,"kb_used_data":3823212,"kb_used_omap":676,"kb_used_meta":1866651,"kb_avail":88681292,"statfs":{"total":96636764160,"available":90809643008,"internally_reserved":0,"allocated":3914969088,"data_stored":7179131127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":692661,"internal_metadata":1911451211},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,9,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:29:03 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.46999999999999997,"15min":0.48599999999999999},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.17199999999999999},"max":{"1min":2.7839999999999998,"5min":4.9480000000000004,"15min":4.9480000000000004},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.52200000000000002,"15min":0.46999999999999997},"min":{"1min":0.218,"5min":0.19400000000000001,"15min":0.186},"max":{"1min":4.5010000000000003,"5min":5.2560000000000002,"15min":5.8479999999999999},"last":0.25800000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:29:03 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.55900000000000005,"15min":0.45800000000000002},"min":{"1min":0.22500000000000001,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":3.6000000000000001,"5min":7.6459999999999999,"15min":7.6459999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.60399999999999998,"15min":0.504},"min":{"1min":0.27700000000000002,"5min":0.253,"15min":0.23799999999999999},"max":{"1min":3.6469999999999998,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.627}]}]},{"osd":0,"up_from":8,"seq":34359739279,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5569432,"kb_used_data":3976528,"kb_used_omap":700,"kb_used_meta":1592195,"kb_avail":88802408,"statfs":{"total":96636764160,"available":90933665792,"internally_reserved":0,"allocated":4071964672,"data_stored":7497106024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":716869,"internal_metadata":1630408635},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,4,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.54400000000000004,"15min":0.79100000000000004},"min":{"1min":0.188,"5min":0.182,"15min":0.16700000000000001},"max":{"1min":0.65200000000000002,"5min":6.3289999999999997,"15min":81.197000000000003},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.44500000000000001,"15min":0.748},"min":{"1min":0.224,"5min":0.19800000000000001,"15min":0.184},"max":{"1min":0.61199999999999999,"5min":6.2229999999999999,"15min":80.334999999999994},"last":0.32700000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.499,"15min":0.50600000000000001},"min":{"1min":0.28899999999999998,"5min":0.216,"15min":0.20200000000000001},"max":{"1min":0.70599999999999996,"5min":6.2859999999999996,"15min":11.176},"last":0.747},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.45100000000000001,"15min":0.52100000000000002},"min":{"1min":0.254,"5min":0.21099999999999999,"15min":0.20000000000000001},"max":{"1min":0.71899999999999997,"5min":4.4260000000000002,"15min":11.093},"last":0.58999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4070109184,"data_stored":7496212480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3913527296,"data_stored":7178801171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7982522368,"data_stored":14673899539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:29:07.874 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:29:07.875 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:29:07.875 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:29:07.875 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:29:09.634 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:29:09.626+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:29:09.636 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:29:09.634+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:29:27.876 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:29:28.441 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:29:28.442 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:29:28.478 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2406,"stamp":"2023-10-20T10:29:27.310344+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27944772099,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":1996562,"num_read_kb":7741817,"num_write":1957117,"num_write_kb":7826719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70891,"ondisk_log_size":70891,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20903028,"kb_used_data":15651160,"kb_used_omap":3123,"kb_used_meta":5248716,"kb_avail":262212492,"statfs":{"total":289910292480,"available":268505591808,"internally_reserved":0,"allocated":16026787840,"data_stored":29406591202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3198639,"internal_metadata":5374685521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,11,15,10,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21307392,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5087,"num_read_kb":20252,"num_write":5225,"num_write_kb":20900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002103"},"pg_stats":[{"pgid":"2.7","version":"130'9913","reported_seq":14763,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:29:24.238109+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:29:24.238109+0000","last_peered":"2023-10-20T10:29:24.238109+0000","last_clean":"2023-10-20T10:29:24.238109+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:29:24.238109+0000","last_undegraded":"2023-10-20T10:29:24.238109+0000","last_fullsized":"2023-10-20T10:29:24.238109+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9913,"log_dups_size":0,"ondisk_log_size":9913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"130'19492","reported_seq":30173,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:29:24.238952+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:29:24.238952+0000","last_peered":"2023-10-20T10:29:24.238952+0000","last_clean":"2023-10-20T10:29:24.238952+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:29:24.238952+0000","last_undegraded":"2023-10-20T10:29:24.238952+0000","last_fullsized":"2023-10-20T10:29:24.238952+0000","mapping_epoch":13,"log_start":"105'9400","ondisk_log_start":"105'9400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":212,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:28:28.174975+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:28:28.174975+0000","last_peered":"2023-10-20T10:28:28.174975+0000","last_clean":"2023-10-20T10:28:28.174975+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:28:28.174975+0000","last_undegraded":"2023-10-20T10:28:28.174975+0000","last_fullsized":"2023-10-20T10:28:28.174975+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"130'9874","reported_seq":15500,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:29:24.238514+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:29:24.238514+0000","last_peered":"2023-10-20T10:29:24.238514+0000","last_clean":"2023-10-20T10:29:24.238514+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:29:24.238514+0000","last_undegraded":"2023-10-20T10:29:24.238514+0000","last_fullsized":"2023-10-20T10:29:24.238514+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9874,"log_dups_size":0,"ondisk_log_size":9874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"130'508734","reported_seq":1008264,"reported_epoch":130,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:29:26.875208+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:29:26.875208+0000","last_peered":"2023-10-20T10:29:26.875208+0000","last_clean":"2023-10-20T10:29:26.875208+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:29:26.875208+0000","last_undegraded":"2023-10-20T10:29:26.875208+0000","last_fullsized":"2023-10-20T10:29:26.875208+0000","mapping_epoch":16,"log_start":"129'498732","ondisk_log_start":"129'498732","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3974,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 582s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7179378688,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":502472,"num_read_kb":1979876,"num_write":502180,"num_write_kb":2008720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":306,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:28:28.161666+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:28:28.161666+0000","last_peered":"2023-10-20T10:28:28.161666+0000","last_clean":"2023-10-20T10:28:28.161666+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:28:28.161666+0000","last_undegraded":"2023-10-20T10:28:28.161666+0000","last_fullsized":"2023-10-20T10:28:28.161666+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"130'472687","reported_seq":936272,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:29:26.906772+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:29:26.906772+0000","last_peered":"2023-10-20T10:29:26.906772+0000","last_clean":"2023-10-20T10:29:26.906772+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:29:26.906772+0000","last_undegraded":"2023-10-20T10:29:26.906772+0000","last_fullsized":"2023-10-20T10:29:26.906772+0000","mapping_epoch":16,"log_start":"129'462642","ondisk_log_start":"129'462642","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6657323008,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":466286,"num_read_kb":1835156,"num_write":465943,"num_write_kb":1863772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6657,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:29:24.238351+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:29:24.238351+0000","last_peered":"2023-10-20T10:29:24.238351+0000","last_clean":"2023-10-20T10:29:24.238351+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:29:24.238351+0000","last_undegraded":"2023-10-20T10:29:24.238351+0000","last_fullsized":"2023-10-20T10:29:24.238351+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"130'501819","reported_seq":994033,"reported_epoch":130,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:29:26.882291+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:29:26.882291+0000","last_peered":"2023-10-20T10:29:26.882291+0000","last_clean":"2023-10-20T10:29:26.882291+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:29:26.882291+0000","last_undegraded":"2023-10-20T10:29:26.882291+0000","last_fullsized":"2023-10-20T10:29:26.882291+0000","mapping_epoch":16,"log_start":"129'491736","ondisk_log_start":"129'491736","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":4407,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 581s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7071490048,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":495164,"num_read_kb":1949240,"num_write":494909,"num_write_kb":1979636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6504,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:29:24.238628+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:29:24.238628+0000","last_peered":"2023-10-20T10:29:24.238628+0000","last_clean":"2023-10-20T10:29:24.238628+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:29:24.238628+0000","last_undegraded":"2023-10-20T10:29:24.238628+0000","last_fullsized":"2023-10-20T10:29:24.238628+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"130'496767","reported_seq":987682,"reported_epoch":130,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:29:26.920070+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:29:26.920070+0000","last_peered":"2023-10-20T10:29:26.920070+0000","last_clean":"2023-10-20T10:29:26.920070+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:29:26.920070+0000","last_undegraded":"2023-10-20T10:29:26.920070+0000","last_fullsized":"2023-10-20T10:29:26.920070+0000","mapping_epoch":16,"log_start":"129'486724","ondisk_log_start":"129'486724","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":294,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 42s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7035985939,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":493969,"num_read_kb":1944824,"num_write":493071,"num_write_kb":1972278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":211,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:28:28.174929+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:28:28.174929+0000","last_peered":"2023-10-20T10:28:28.174929+0000","last_clean":"2023-10-20T10:28:28.174929+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:28:28.174929+0000","last_undegraded":"2023-10-20T10:28:28.174929+0000","last_fullsized":"2023-10-20T10:28:28.174929+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":304,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-20T10:28:01.391495+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:28:01.391495+0000","last_peered":"2023-10-20T10:28:01.391495+0000","last_clean":"2023-10-20T10:28:01.391495+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:28:01.391495+0000","last_undegraded":"2023-10-20T10:28:01.391495+0000","last_fullsized":"2023-10-20T10:28:01.391495+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27944177683,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1957891,"num_read_kb":7709096,"num_write":1956103,"num_write_kb":7824406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16023339008,"data_stored":29406093350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40173,"ondisk_log_size":40173,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":30619,"ondisk_log_size":30619,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739281,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9975956,"kb_used_data":7823436,"kb_used_omap":1745,"kb_used_meta":2150766,"kb_avail":84395884,"statfs":{"total":96636764160,"available":86421385216,"internally_reserved":0,"allocated":8011198464,"data_stored":14701698435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1787431,"internal_metadata":2202384857},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:29:10 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.45100000000000001,"15min":0.47899999999999998},"min":{"1min":0.218,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":6.2430000000000003,"5min":6.2430000000000003,"15min":8.2449999999999992},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.51600000000000001,"15min":0.49299999999999999},"min":{"1min":0.20999999999999999,"5min":0.20799999999999999,"15min":0.18099999999999999},"max":{"1min":0.56399999999999995,"5min":5.048,"15min":6.8239999999999998},"last":0.29899999999999999}]},{"osd":1,"last update":"Fri Oct 20 10:29:10 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.60599999999999998,"15min":0.60399999999999998},"min":{"1min":0.23400000000000001,"5min":0.186,"15min":0.186},"max":{"1min":6.0800000000000001,"5min":6.1509999999999998,"15min":8.282},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.52800000000000002,"15min":0.48299999999999998},"min":{"1min":0.27800000000000002,"5min":0.22900000000000001,"15min":0.214},"max":{"1min":6.109,"5min":6.109,"15min":6.109},"last":0.26600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739281,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5307948,"kb_used_data":3837600,"kb_used_omap":678,"kb_used_meta":1469657,"kb_avail":89063892,"statfs":{"total":96636764160,"available":91201425408,"internally_reserved":0,"allocated":3929702400,"data_stored":7193864439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":694339,"internal_metadata":1504929725},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,13,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:29:03 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.46999999999999997,"15min":0.48599999999999999},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.17199999999999999},"max":{"1min":2.7839999999999998,"5min":4.9480000000000004,"15min":4.9480000000000004},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.52200000000000002,"15min":0.46999999999999997},"min":{"1min":0.218,"5min":0.19400000000000001,"15min":0.186},"max":{"1min":4.5010000000000003,"5min":5.2560000000000002,"15min":5.8479999999999999},"last":0.251}]},{"osd":2,"last update":"Fri Oct 20 10:29:03 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.55900000000000005,"15min":0.45800000000000002},"min":{"1min":0.22500000000000001,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":3.6000000000000001,"5min":7.6459999999999999,"15min":7.6459999999999999},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.60399999999999998,"15min":0.504},"min":{"1min":0.27700000000000002,"5min":0.253,"15min":0.23799999999999999},"max":{"1min":3.6469999999999998,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.35399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739283,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5619124,"kb_used_data":3990124,"kb_used_omap":700,"kb_used_meta":1628291,"kb_avail":88752716,"statfs":{"total":96636764160,"available":90882781184,"internally_reserved":0,"allocated":4085886976,"data_stored":7511028328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":716869,"internal_metadata":1667370939},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,6,7,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.54400000000000004,"15min":0.79100000000000004},"min":{"1min":0.188,"5min":0.182,"15min":0.16700000000000001},"max":{"1min":0.65200000000000002,"5min":6.3289999999999997,"15min":81.197000000000003},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.44500000000000001,"15min":0.748},"min":{"1min":0.224,"5min":0.19800000000000001,"15min":0.184},"max":{"1min":0.61199999999999999,"5min":6.2229999999999999,"15min":80.334999999999994},"last":0.20699999999999999}]},{"osd":2,"last update":"Fri Oct 20 10:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.499,"15min":0.50600000000000001},"min":{"1min":0.28899999999999998,"5min":0.216,"15min":0.20200000000000001},"max":{"1min":0.70599999999999996,"5min":6.2859999999999996,"15min":11.176},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.45100000000000001,"15min":0.52100000000000002},"min":{"1min":0.254,"5min":0.21099999999999999,"15min":0.20000000000000001},"max":{"1min":0.71899999999999997,"5min":4.4260000000000002,"15min":11.093},"last":0.27100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4084080640,"data_stored":7510183936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3928129536,"data_stored":7193403411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8011128832,"data_stored":14702506003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:29:28.480 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:29:28.481 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:29:28.481 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:29:28.482 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:29:48.483 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:29:48.931 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:29:48.933 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:29:48.961 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2417,"stamp":"2023-10-20T10:29:47.313963+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27973612035,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2003275,"num_read_kb":7768569,"num_write":1964186,"num_write_kb":7854995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71006,"ondisk_log_size":71006,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20432088,"kb_used_data":15707772,"kb_used_omap":3125,"kb_used_meta":4721162,"kb_avail":262683432,"statfs":{"total":289910292480,"available":268987834368,"internally_reserved":0,"allocated":16084758528,"data_stored":29464538205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3200319,"internal_metadata":4834470593},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,5,12,11,12,12,7,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14221312,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3262,"num_read_kb":13000,"num_write":3486,"num_write_kb":13944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001693"},"pg_stats":[{"pgid":"2.7","version":"131'9934","reported_seq":14797,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:29:44.838965+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:29:44.838965+0000","last_peered":"2023-10-20T10:29:44.838965+0000","last_clean":"2023-10-20T10:29:44.838965+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:29:44.838965+0000","last_undegraded":"2023-10-20T10:29:44.838965+0000","last_fullsized":"2023-10-20T10:29:44.838965+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9934,"log_dups_size":0,"ondisk_log_size":9934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"131'19534","reported_seq":30241,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:29:44.860097+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:29:44.860097+0000","last_peered":"2023-10-20T10:29:44.860097+0000","last_clean":"2023-10-20T10:29:44.860097+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:29:44.860097+0000","last_undegraded":"2023-10-20T10:29:44.860097+0000","last_fullsized":"2023-10-20T10:29:44.860097+0000","mapping_epoch":13,"log_start":"107'9500","ondisk_log_start":"107'9500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":213,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:29:06.831018+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:29:06.831018+0000","last_peered":"2023-10-20T10:29:06.831018+0000","last_clean":"2023-10-20T10:29:06.831018+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:29:06.831018+0000","last_undegraded":"2023-10-20T10:29:06.831018+0000","last_fullsized":"2023-10-20T10:29:06.831018+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"131'9895","reported_seq":15534,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:29:44.848741+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:29:44.848741+0000","last_peered":"2023-10-20T10:29:44.848741+0000","last_clean":"2023-10-20T10:29:44.848741+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:29:44.848741+0000","last_undegraded":"2023-10-20T10:29:44.848741+0000","last_fullsized":"2023-10-20T10:29:44.848741+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9895,"log_dups_size":0,"ondisk_log_size":9895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"131'510631","reported_seq":1011905,"reported_epoch":131,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:29:47.009925+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:29:47.009925+0000","last_peered":"2023-10-20T10:29:47.009925+0000","last_clean":"2023-10-20T10:29:47.009925+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:29:47.009925+0000","last_undegraded":"2023-10-20T10:29:47.009925+0000","last_fullsized":"2023-10-20T10:29:47.009925+0000","mapping_epoch":16,"log_start":"129'500532","ondisk_log_start":"129'500532","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":4111,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 603s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7186690048,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":504204,"num_read_kb":1986780,"num_write":503970,"num_write_kb":2015880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":307,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:29:06.751943+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:29:06.751943+0000","last_peered":"2023-10-20T10:29:06.751943+0000","last_clean":"2023-10-20T10:29:06.751943+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:29:06.751943+0000","last_undegraded":"2023-10-20T10:29:06.751943+0000","last_fullsized":"2023-10-20T10:29:06.751943+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"131'474434","reported_seq":939585,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:29:46.990269+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:29:46.990269+0000","last_peered":"2023-10-20T10:29:46.990269+0000","last_clean":"2023-10-20T10:29:46.990269+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:29:46.990269+0000","last_undegraded":"2023-10-20T10:29:46.990269+0000","last_fullsized":"2023-10-20T10:29:46.990269+0000","mapping_epoch":16,"log_start":"129'464342","ondisk_log_start":"129'464342","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6664441856,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":467851,"num_read_kb":1841400,"num_write":467690,"num_write_kb":1870760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6674,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:29:44.718706+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:29:44.718706+0000","last_peered":"2023-10-20T10:29:44.718706+0000","last_clean":"2023-10-20T10:29:44.718706+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:29:44.718706+0000","last_undegraded":"2023-10-20T10:29:44.718706+0000","last_fullsized":"2023-10-20T10:29:44.718706+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"131'503736","reported_seq":997698,"reported_epoch":131,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:29:47.013629+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:29:47.013629+0000","last_peered":"2023-10-20T10:29:47.013629+0000","last_clean":"2023-10-20T10:29:47.013629+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:29:47.013629+0000","last_undegraded":"2023-10-20T10:29:47.013629+0000","last_fullsized":"2023-10-20T10:29:47.013629+0000","mapping_epoch":16,"log_start":"129'493636","ondisk_log_start":"129'493636","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":4570,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 602s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7078776832,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":496897,"num_read_kb":1956144,"num_write":496701,"num_write_kb":1986804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6521,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:29:44.720216+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:29:44.720216+0000","last_peered":"2023-10-20T10:29:44.720216+0000","last_clean":"2023-10-20T10:29:44.720216+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:29:44.720216+0000","last_undegraded":"2023-10-20T10:29:44.720216+0000","last_fullsized":"2023-10-20T10:29:44.720216+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"131'498637","reported_seq":991252,"reported_epoch":131,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:29:47.002411+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:29:47.002411+0000","last_peered":"2023-10-20T10:29:47.002411+0000","last_clean":"2023-10-20T10:29:47.002411+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:29:47.002411+0000","last_undegraded":"2023-10-20T10:29:47.002411+0000","last_fullsized":"2023-10-20T10:29:47.002411+0000","mapping_epoch":16,"log_start":"129'488624","ondisk_log_start":"129'488624","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":464,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 62s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7043108883,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":495652,"num_read_kb":1951524,"num_write":494811,"num_write_kb":1979238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":212,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:29:06.830969+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:29:06.830969+0000","last_peered":"2023-10-20T10:29:06.830969+0000","last_clean":"2023-10-20T10:29:06.830969+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:29:06.830969+0000","last_undegraded":"2023-10-20T10:29:06.830969+0000","last_fullsized":"2023-10-20T10:29:06.830969+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":305,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:28:28.180880+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:28:28.180880+0000","last_peered":"2023-10-20T10:28:28.180880+0000","last_clean":"2023-10-20T10:28:28.180880+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:28:28.180880+0000","last_undegraded":"2023-10-20T10:28:28.180880+0000","last_fullsized":"2023-10-20T10:28:28.180880+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27973017619,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1964604,"num_read_kb":7735848,"num_write":1963172,"num_write_kb":7852682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16081211392,"data_stored":29463965734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40304,"ondisk_log_size":40304,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":30603,"ondisk_log_size":30603,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739285,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9386524,"kb_used_data":7851728,"kb_used_omap":1747,"kb_used_meta":1533036,"kb_avail":84985316,"statfs":{"total":96636764160,"available":87024963584,"internally_reserved":0,"allocated":8040169472,"data_stored":14730661548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1789111,"internal_metadata":1569829705},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:29:10 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.45100000000000001,"15min":0.47899999999999998},"min":{"1min":0.218,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":6.2430000000000003,"5min":6.2430000000000003,"15min":8.2449999999999992},"last":0.252},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.51600000000000001,"15min":0.49299999999999999},"min":{"1min":0.20999999999999999,"5min":0.20799999999999999,"15min":0.18099999999999999},"max":{"1min":0.56399999999999995,"5min":5.048,"15min":6.8239999999999998},"last":0.28899999999999998}]},{"osd":1,"last update":"Fri Oct 20 10:29:10 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.60599999999999998,"15min":0.60399999999999998},"min":{"1min":0.23400000000000001,"5min":0.186,"15min":0.186},"max":{"1min":6.0800000000000001,"5min":6.1509999999999998,"15min":8.282},"last":0.223},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.52800000000000002,"15min":0.48299999999999998},"min":{"1min":0.27800000000000002,"5min":0.22900000000000001,"15min":0.214},"max":{"1min":6.109,"5min":6.109,"15min":6.109},"last":0.26800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739285,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5358184,"kb_used_data":3851744,"kb_used_omap":678,"kb_used_meta":1505753,"kb_avail":89013656,"statfs":{"total":96636764160,"available":91149983744,"internally_reserved":0,"allocated":3944185856,"data_stored":7208340000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":694339,"internal_metadata":1541892029},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,4,3,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:29:03 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.46999999999999997,"15min":0.48599999999999999},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.17199999999999999},"max":{"1min":2.7839999999999998,"5min":4.9480000000000004,"15min":4.9480000000000004},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.52200000000000002,"15min":0.46999999999999997},"min":{"1min":0.218,"5min":0.19400000000000001,"15min":0.186},"max":{"1min":4.5010000000000003,"5min":5.2560000000000002,"15min":5.8479999999999999},"last":0.23999999999999999}]},{"osd":2,"last update":"Fri Oct 20 10:29:03 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.55900000000000005,"15min":0.45800000000000002},"min":{"1min":0.22500000000000001,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":3.6000000000000001,"5min":7.6459999999999999,"15min":7.6459999999999999},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.60399999999999998,"15min":0.504},"min":{"1min":0.27700000000000002,"5min":0.253,"15min":0.23799999999999999},"max":{"1min":3.6469999999999998,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.32700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739287,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5687380,"kb_used_data":4004300,"kb_used_omap":700,"kb_used_meta":1682371,"kb_avail":88684460,"statfs":{"total":96636764160,"available":90812887040,"internally_reserved":0,"allocated":4100403200,"data_stored":7525536657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":716869,"internal_metadata":1722748859},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,6,7,4,9,7,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.54400000000000004,"15min":0.79100000000000004},"min":{"1min":0.188,"5min":0.182,"15min":0.16700000000000001},"max":{"1min":0.65200000000000002,"5min":6.3289999999999997,"15min":81.197000000000003},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.44500000000000001,"15min":0.748},"min":{"1min":0.224,"5min":0.19800000000000001,"15min":0.184},"max":{"1min":0.61199999999999999,"5min":6.2229999999999999,"15min":80.334999999999994},"last":0.42999999999999999}]},{"osd":2,"last update":"Fri Oct 20 10:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.499,"15min":0.50600000000000001},"min":{"1min":0.28899999999999998,"5min":0.216,"15min":0.20200000000000001},"max":{"1min":0.70599999999999996,"5min":6.2859999999999996,"15min":11.176},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.45100000000000001,"15min":0.52100000000000002},"min":{"1min":0.254,"5min":0.21099999999999999,"15min":0.20000000000000001},"max":{"1min":0.71899999999999997,"5min":4.4260000000000002,"15min":11.093},"last":0.41799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4098756608,"data_stored":7524859904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3942404096,"data_stored":7207677971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8040050688,"data_stored":14731427859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:29:48.963 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:29:48.964 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:29:48.964 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:29:48.965 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:29:51.608 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:29:51.606+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:29:51.624 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:29:51.622+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:30:04.458 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:30:04.449+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:30:08.966 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:30:09.471 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:30:09.472 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:30:09.503 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2428,"stamp":"2023-10-20T10:30:09.317905+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28002181635,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2009982,"num_read_kb":7795297,"num_write":1971190,"num_write_kb":7883011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70886,"ondisk_log_size":70886,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20661284,"kb_used_data":15763916,"kb_used_omap":3125,"kb_used_meta":4894218,"kb_avail":262454236,"statfs":{"total":289910292480,"available":268753137664,"internally_reserved":0,"allocated":16142249984,"data_stored":29522029661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3200319,"internal_metadata":5011679937},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,9,9,6,9,18],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14462976,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3311,"num_read_kb":13200,"num_write":3544,"num_write_kb":14176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002220"},"pg_stats":[{"pgid":"2.7","version":"131'9969","reported_seq":14845,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:30:06.429813+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:30:06.429813+0000","last_peered":"2023-10-20T10:30:06.429813+0000","last_clean":"2023-10-20T10:30:06.429813+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:30:06.429813+0000","last_undegraded":"2023-10-20T10:30:06.429813+0000","last_fullsized":"2023-10-20T10:30:06.429813+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":9969,"log_dups_size":0,"ondisk_log_size":9969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"131'19601","reported_seq":30332,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:30:06.475170+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:30:06.475170+0000","last_peered":"2023-10-20T10:30:06.475170+0000","last_clean":"2023-10-20T10:30:06.475170+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:30:06.475170+0000","last_undegraded":"2023-10-20T10:30:06.475170+0000","last_fullsized":"2023-10-20T10:30:06.475170+0000","mapping_epoch":13,"log_start":"107'9600","ondisk_log_start":"107'9600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":213,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:29:06.831018+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:29:06.831018+0000","last_peered":"2023-10-20T10:29:06.831018+0000","last_clean":"2023-10-20T10:29:06.831018+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:29:06.831018+0000","last_undegraded":"2023-10-20T10:29:06.831018+0000","last_fullsized":"2023-10-20T10:29:06.831018+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"131'9930","reported_seq":15581,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:30:06.434627+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:30:06.434627+0000","last_peered":"2023-10-20T10:30:06.434627+0000","last_clean":"2023-10-20T10:30:06.434627+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:30:06.434627+0000","last_undegraded":"2023-10-20T10:30:06.434627+0000","last_fullsized":"2023-10-20T10:30:06.434627+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9930,"log_dups_size":0,"ondisk_log_size":9930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"131'512568","reported_seq":1015617,"reported_epoch":131,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:30:07.119591+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:30:07.119591+0000","last_peered":"2023-10-20T10:30:07.119591+0000","last_clean":"2023-10-20T10:30:07.119591+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:30:07.119591+0000","last_undegraded":"2023-10-20T10:30:07.119591+0000","last_fullsized":"2023-10-20T10:30:07.119591+0000","mapping_epoch":16,"log_start":"129'502532","ondisk_log_start":"129'502532","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":4280,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 623s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7194071040,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":505967,"num_read_kb":1993804,"num_write":505776,"num_write_kb":2023104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":307,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:29:06.751943+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:29:06.751943+0000","last_peered":"2023-10-20T10:29:06.751943+0000","last_clean":"2023-10-20T10:29:06.751943+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:29:06.751943+0000","last_undegraded":"2023-10-20T10:29:06.751943+0000","last_fullsized":"2023-10-20T10:29:06.751943+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"131'476143","reported_seq":942906,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:30:07.098111+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:30:07.098111+0000","last_peered":"2023-10-20T10:30:07.098111+0000","last_clean":"2023-10-20T10:30:07.098111+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:30:07.098111+0000","last_undegraded":"2023-10-20T10:30:07.098111+0000","last_fullsized":"2023-10-20T10:30:07.098111+0000","mapping_epoch":16,"log_start":"129'466142","ondisk_log_start":"129'466142","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6671417344,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":469468,"num_read_kb":1847848,"num_write":469399,"num_write_kb":1877596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6690,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:30:05.202790+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:30:05.202790+0000","last_peered":"2023-10-20T10:30:05.202790+0000","last_clean":"2023-10-20T10:30:05.202790+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:30:05.202790+0000","last_undegraded":"2023-10-20T10:30:05.202790+0000","last_fullsized":"2023-10-20T10:30:05.202790+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"131'505619","reported_seq":1001271,"reported_epoch":131,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:30:07.122284+0000","last_change":"2023-10-20T10:19:44.927396+0000","last_active":"2023-10-20T10:30:07.122284+0000","last_peered":"2023-10-20T10:30:07.122284+0000","last_clean":"2023-10-20T10:30:07.122284+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:30:07.122284+0000","last_undegraded":"2023-10-20T10:30:07.122284+0000","last_fullsized":"2023-10-20T10:30:07.122284+0000","mapping_epoch":16,"log_start":"129'495536","ondisk_log_start":"129'495536","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":4706,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 622s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7086022656,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":498571,"num_read_kb":1962816,"num_write":498478,"num_write_kb":1993912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6538,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:30:05.203295+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:30:05.203295+0000","last_peered":"2023-10-20T10:30:05.203295+0000","last_clean":"2023-10-20T10:30:05.203295+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:30:05.203295+0000","last_undegraded":"2023-10-20T10:30:05.203295+0000","last_fullsized":"2023-10-20T10:30:05.203295+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"131'500451","reported_seq":994732,"reported_epoch":131,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:30:07.124526+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:30:07.124526+0000","last_peered":"2023-10-20T10:30:07.124526+0000","last_clean":"2023-10-20T10:30:07.124526+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:30:07.124526+0000","last_undegraded":"2023-10-20T10:30:07.124526+0000","last_fullsized":"2023-10-20T10:30:07.124526+0000","mapping_epoch":16,"log_start":"129'490424","ondisk_log_start":"129'490424","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":602,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 82s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7050076179,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":497305,"num_read_kb":1958108,"num_write":496523,"num_write_kb":1986086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":212,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:29:06.830969+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:29:06.830969+0000","last_peered":"2023-10-20T10:29:06.830969+0000","last_clean":"2023-10-20T10:29:06.830969+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:29:06.830969+0000","last_undegraded":"2023-10-20T10:29:06.830969+0000","last_fullsized":"2023-10-20T10:29:06.830969+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":306,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:29:38.209891+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:29:38.209891+0000","last_peered":"2023-10-20T10:29:38.209891+0000","last_clean":"2023-10-20T10:29:38.209891+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:29:38.209891+0000","last_undegraded":"2023-10-20T10:29:38.209891+0000","last_fullsized":"2023-10-20T10:29:38.209891+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28001587219,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1971311,"num_read_kb":7762576,"num_write":1970176,"num_write_kb":7880698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16138653696,"data_stored":29521408038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40147,"ondisk_log_size":40147,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":30640,"ondisk_log_size":30640,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739289,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9535104,"kb_used_data":7879864,"kb_used_omap":1747,"kb_used_meta":1653484,"kb_avail":84836736,"statfs":{"total":96636764160,"available":86872817664,"internally_reserved":0,"allocated":8068980736,"data_stored":14759472812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1789111,"internal_metadata":1693168457},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:29:10 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.45100000000000001,"15min":0.47899999999999998},"min":{"1min":0.218,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":6.2430000000000003,"5min":6.2430000000000003,"15min":8.2449999999999992},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.51600000000000001,"15min":0.49299999999999999},"min":{"1min":0.20999999999999999,"5min":0.20799999999999999,"15min":0.18099999999999999},"max":{"1min":0.56399999999999995,"5min":5.048,"15min":6.8239999999999998},"last":0.38700000000000001}]},{"osd":1,"last update":"Fri Oct 20 10:29:10 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.60599999999999998,"15min":0.60399999999999998},"min":{"1min":0.23400000000000001,"5min":0.186,"15min":0.186},"max":{"1min":6.0800000000000001,"5min":6.1509999999999998,"15min":8.282},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.52800000000000002,"15min":0.48299999999999998},"min":{"1min":0.27800000000000002,"5min":0.22900000000000001,"15min":0.214},"max":{"1min":6.109,"5min":6.109,"15min":6.109},"last":0.36099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739289,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5403716,"kb_used_data":3865340,"kb_used_omap":678,"kb_used_meta":1537689,"kb_avail":88968124,"statfs":{"total":96636764160,"available":91103358976,"internally_reserved":0,"allocated":3958108160,"data_stored":7222262304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":694339,"internal_metadata":1574594493},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,3,3,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:29:03 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.46999999999999997,"15min":0.48599999999999999},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.17199999999999999},"max":{"1min":2.7839999999999998,"5min":4.9480000000000004,"15min":4.9480000000000004},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.52200000000000002,"15min":0.46999999999999997},"min":{"1min":0.218,"5min":0.19400000000000001,"15min":0.186},"max":{"1min":4.5010000000000003,"5min":5.2560000000000002,"15min":5.8479999999999999},"last":0.27100000000000002}]},{"osd":2,"last update":"Fri Oct 20 10:29:03 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.55900000000000005,"15min":0.45800000000000002},"min":{"1min":0.22500000000000001,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":3.6000000000000001,"5min":7.6459999999999999,"15min":7.6459999999999999},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.60399999999999998,"15min":0.504},"min":{"1min":0.27700000000000002,"5min":0.253,"15min":0.23799999999999999},"max":{"1min":3.6469999999999998,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.32600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739291,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5722464,"kb_used_data":4018712,"kb_used_omap":700,"kb_used_meta":1703043,"kb_avail":88649376,"statfs":{"total":96636764160,"available":90776961024,"internally_reserved":0,"allocated":4115161088,"data_stored":7540294545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":716869,"internal_metadata":1743916987},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,2,4,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:30:04 2023","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":0.64100000000000001,"15min":0.84599999999999997},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":6.5449999999999999,"5min":6.5449999999999999,"15min":81.197000000000003},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.502,"15min":0.78000000000000003},"min":{"1min":0.20699999999999999,"5min":0.19800000000000001,"15min":0.184},"max":{"1min":6.024,"5min":6.2229999999999999,"15min":80.334999999999994},"last":0.27800000000000002}]},{"osd":2,"last update":"Fri Oct 20 10:30:04 2023","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":0.60899999999999999,"15min":0.52900000000000003},"min":{"1min":0.252,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":6.3029999999999999,"5min":6.3029999999999999,"15min":11.176},"last":0.317},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.47899999999999998,"15min":0.501},"min":{"1min":0.24299999999999999,"5min":0.21099999999999999,"15min":0.20000000000000001},"max":{"1min":4.0220000000000002,"5min":4.4260000000000002,"15min":11.093},"last":0.26100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4113424384,"data_stored":7539527680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3956420608,"data_stored":7221694483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8068808704,"data_stored":14760185875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:30:09.505 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:30:09.505 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:30:09.506 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:30:09.506 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:30:11.036 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:30:11.021+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:30:11.050 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:30:11.041+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T10:30:11.051 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:30:11.041+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:30:11.051 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:30:11.041+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T10:30:11.051 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:30:11.045+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T10:30:11.051 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:30:11.045+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:30:11.052 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:30:11.045+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:30:29.507 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:30:29.954 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:30:29.955 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:30:29.995 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2438,"stamp":"2023-10-20T10:30:29.321204+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28031566339,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2016703,"num_read_kb":7822117,"num_write":1978385,"num_write_kb":7911791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71171,"ondisk_log_size":71171,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20931600,"kb_used_data":15820920,"kb_used_omap":3125,"kb_used_meta":5107530,"kb_avail":262183920,"statfs":{"total":289910292480,"available":268476334080,"internally_reserved":0,"allocated":16200622080,"data_stored":29580401757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3200319,"internal_metadata":5230111425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,7,10,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14958592,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3413,"num_read_kb":13628,"num_write":3659,"num_write_kb":14636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001997"},"pg_stats":[{"pgid":"2.7","version":"131'10002","reported_seq":14890,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:30:25.679367+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:30:25.679367+0000","last_peered":"2023-10-20T10:30:25.679367+0000","last_clean":"2023-10-20T10:30:25.679367+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:30:25.679367+0000","last_undegraded":"2023-10-20T10:30:25.679367+0000","last_fullsized":"2023-10-20T10:30:25.679367+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"131'19667","reported_seq":30422,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:30:25.678945+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:30:25.678945+0000","last_peered":"2023-10-20T10:30:25.678945+0000","last_clean":"2023-10-20T10:30:25.678945+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:30:25.678945+0000","last_undegraded":"2023-10-20T10:30:25.678945+0000","last_fullsized":"2023-10-20T10:30:25.678945+0000","mapping_epoch":13,"log_start":"107'9600","ondisk_log_start":"107'9600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":213,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:29:06.831018+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:29:06.831018+0000","last_peered":"2023-10-20T10:29:06.831018+0000","last_clean":"2023-10-20T10:29:06.831018+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:29:06.831018+0000","last_undegraded":"2023-10-20T10:29:06.831018+0000","last_fullsized":"2023-10-20T10:29:06.831018+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"131'9963","reported_seq":15626,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:30:25.678198+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:30:25.678198+0000","last_peered":"2023-10-20T10:30:25.678198+0000","last_clean":"2023-10-20T10:30:25.678198+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:30:25.678198+0000","last_undegraded":"2023-10-20T10:30:25.678198+0000","last_fullsized":"2023-10-20T10:30:25.678198+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9963,"log_dups_size":0,"ondisk_log_size":9963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"131'514547","reported_seq":1019396,"reported_epoch":131,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:30:27.254947+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:30:27.254947+0000","last_peered":"2023-10-20T10:30:27.254947+0000","last_clean":"2023-10-20T10:30:27.254947+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:30:27.254947+0000","last_undegraded":"2023-10-20T10:30:27.254947+0000","last_fullsized":"2023-10-20T10:30:27.254947+0000","mapping_epoch":16,"log_start":"130'504532","ondisk_log_start":"130'504532","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":4434,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 643s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7201693696,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":507761,"num_read_kb":2000960,"num_write":507640,"num_write_kb":2030560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":307,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:29:06.751943+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:29:06.751943+0000","last_peered":"2023-10-20T10:29:06.751943+0000","last_clean":"2023-10-20T10:29:06.751943+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:29:06.751943+0000","last_undegraded":"2023-10-20T10:29:06.751943+0000","last_fullsized":"2023-10-20T10:29:06.751943+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"131'477835","reported_seq":946164,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:30:27.165150+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:30:27.165150+0000","last_peered":"2023-10-20T10:30:27.165150+0000","last_clean":"2023-10-20T10:30:27.165150+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:30:27.165150+0000","last_undegraded":"2023-10-20T10:30:27.165150+0000","last_fullsized":"2023-10-20T10:30:27.165150+0000","mapping_epoch":16,"log_start":"130'467742","ondisk_log_start":"130'467742","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6678327296,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":471030,"num_read_kb":1854084,"num_write":471091,"num_write_kb":1884364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6706,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:30:25.678223+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:30:25.678223+0000","last_peered":"2023-10-20T10:30:25.678223+0000","last_clean":"2023-10-20T10:30:25.678223+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:30:25.678223+0000","last_undegraded":"2023-10-20T10:30:25.678223+0000","last_fullsized":"2023-10-20T10:30:25.678223+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"131'507536","reported_seq":1004932,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:30:27.224163+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:30:27.224163+0000","last_peered":"2023-10-20T10:30:27.224163+0000","last_clean":"2023-10-20T10:30:27.224163+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:30:27.224163+0000","last_undegraded":"2023-10-20T10:30:27.224163+0000","last_fullsized":"2023-10-20T10:30:27.224163+0000","mapping_epoch":16,"log_start":"130'497436","ondisk_log_start":"130'497436","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7093448704,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":500302,"num_read_kb":1969724,"num_write":500298,"num_write_kb":2001192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6554,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:30:25.691876+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:30:25.691876+0000","last_peered":"2023-10-20T10:30:25.691876+0000","last_clean":"2023-10-20T10:30:25.691876+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:30:25.691876+0000","last_undegraded":"2023-10-20T10:30:25.691876+0000","last_fullsized":"2023-10-20T10:30:25.691876+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"131'502416","reported_seq":998338,"reported_epoch":131,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:30:27.175819+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:30:27.175819+0000","last_peered":"2023-10-20T10:30:27.175819+0000","last_clean":"2023-10-20T10:30:27.175819+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:30:27.175819+0000","last_undegraded":"2023-10-20T10:30:27.175819+0000","last_fullsized":"2023-10-20T10:30:27.175819+0000","mapping_epoch":16,"log_start":"130'492324","ondisk_log_start":"130'492324","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":791,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 102s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7057502227,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":498939,"num_read_kb":1964628,"num_write":498342,"num_write_kb":1993362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":212,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:29:06.830969+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:29:06.830969+0000","last_peered":"2023-10-20T10:29:06.830969+0000","last_clean":"2023-10-20T10:29:06.830969+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:29:06.830969+0000","last_undegraded":"2023-10-20T10:29:06.830969+0000","last_fullsized":"2023-10-20T10:29:06.830969+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":306,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:29:38.209891+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:29:38.209891+0000","last_peered":"2023-10-20T10:29:38.209891+0000","last_clean":"2023-10-20T10:29:38.209891+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:29:38.209891+0000","last_undegraded":"2023-10-20T10:29:38.209891+0000","last_fullsized":"2023-10-20T10:29:38.209891+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28030971923,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1978032,"num_read_kb":7789396,"num_write":1977371,"num_write_kb":7909478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16197431296,"data_stored":29580185638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40300,"ondisk_log_size":40300,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":30772,"ondisk_log_size":30772,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739293,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9661980,"kb_used_data":7908180,"kb_used_omap":1747,"kb_used_meta":1752044,"kb_avail":84709860,"statfs":{"total":96636764160,"available":86742896640,"internally_reserved":0,"allocated":8097976320,"data_stored":14788468396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1789111,"internal_metadata":1794093897},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.45700000000000002,"15min":0.47199999999999998},"min":{"1min":0.246,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":0.496,"5min":6.2430000000000003,"15min":8.2449999999999992},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.52000000000000002,"15min":0.48699999999999999},"min":{"1min":0.23599999999999999,"5min":0.20799999999999999,"15min":0.18099999999999999},"max":{"1min":0.73699999999999999,"5min":5.048,"15min":6.8239999999999998},"last":0.34899999999999998}]},{"osd":1,"last update":"Fri Oct 20 10:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.63400000000000001,"15min":0.60699999999999998},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":3.0790000000000002,"5min":6.1509999999999998,"15min":8.282},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.53900000000000003,"15min":0.47499999999999998},"min":{"1min":0.23799999999999999,"5min":0.23400000000000001,"15min":0.214},"max":{"1min":0.72899999999999998,"5min":6.109,"15min":6.109},"last":0.30499999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739293,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5460468,"kb_used_data":3879468,"kb_used_omap":678,"kb_used_meta":1580313,"kb_avail":88911372,"statfs":{"total":96636764160,"available":91045244928,"internally_reserved":0,"allocated":3972575232,"data_stored":7236729376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":694339,"internal_metadata":1618241469},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,4,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.48499999999999999,"15min":0.48299999999999998},"min":{"1min":0.26700000000000002,"5min":0.20000000000000001,"15min":0.17199999999999999},"max":{"1min":1.548,"5min":4.9480000000000004,"15min":4.9480000000000004},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.27300000000000002,"5min":0.46800000000000003,"15min":0.45000000000000001},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.186},"max":{"1min":0.40400000000000003,"5min":5.0049999999999999,"15min":5.8479999999999999},"last":0.30199999999999999}]},{"osd":2,"last update":"Fri Oct 20 10:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.56299999999999994,"15min":0.46100000000000002},"min":{"1min":0.23699999999999999,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":0.60299999999999998,"5min":7.6459999999999999,"15min":7.6459999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.60699999999999998,"15min":0.50700000000000001},"min":{"1min":0.29699999999999999,"5min":0.253,"15min":0.23799999999999999},"max":{"1min":0.627,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.39000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739295,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5809152,"kb_used_data":4033272,"kb_used_omap":700,"kb_used_meta":1775171,"kb_avail":88562688,"statfs":{"total":96636764160,"available":90688192512,"internally_reserved":0,"allocated":4130070528,"data_stored":7555203985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":716869,"internal_metadata":1817776059},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:30:04 2023","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":0.64100000000000001,"15min":0.84599999999999997},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":6.5449999999999999,"5min":6.5449999999999999,"15min":81.197000000000003},"last":0.315},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.502,"15min":0.78000000000000003},"min":{"1min":0.20699999999999999,"5min":0.19800000000000001,"15min":0.184},"max":{"1min":6.024,"5min":6.2229999999999999,"15min":80.334999999999994},"last":0.503}]},{"osd":2,"last update":"Fri Oct 20 10:30:04 2023","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":0.60899999999999999,"15min":0.52900000000000003},"min":{"1min":0.252,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":6.3029999999999999,"5min":6.3029999999999999,"15min":11.176},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.47899999999999998,"15min":0.501},"min":{"1min":0.24299999999999999,"5min":0.21099999999999999,"15min":0.20000000000000001},"max":{"1min":4.0220000000000002,"5min":4.4260000000000002,"15min":11.093},"last":0.42399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4128526336,"data_stored":7554629632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3970797568,"data_stored":7236071443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8098107392,"data_stored":14789484563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:30:29.997 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:30:29.997 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:30:29.998 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:30:49.999 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:30:50.415 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:30:50.416 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:30:50.445 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2448,"stamp":"2023-10-20T10:30:49.325621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28062044675,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2023996,"num_read_kb":7851225,"num_write":1985843,"num_write_kb":7941623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71051,"ondisk_log_size":71051,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21161916,"kb_used_data":15882148,"kb_used_omap":3125,"kb_used_meta":5276618,"kb_avail":261953604,"statfs":{"total":289910292480,"available":268240490496,"internally_reserved":0,"allocated":16263319552,"data_stored":29643099229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3200319,"internal_metadata":5403257537},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,1,9,8,10,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15220736,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3664,"num_read_kb":14632,"num_write":3719,"num_write_kb":14876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003193"},"pg_stats":[{"pgid":"2.7","version":"131'10032","reported_seq":14932,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:30:46.161978+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:30:46.161978+0000","last_peered":"2023-10-20T10:30:46.161978+0000","last_clean":"2023-10-20T10:30:46.161978+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:30:46.161978+0000","last_undegraded":"2023-10-20T10:30:46.161978+0000","last_fullsized":"2023-10-20T10:30:46.161978+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"131'19727","reported_seq":30506,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:30:46.162600+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:30:46.162600+0000","last_peered":"2023-10-20T10:30:46.162600+0000","last_clean":"2023-10-20T10:30:46.162600+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:30:46.162600+0000","last_undegraded":"2023-10-20T10:30:46.162600+0000","last_fullsized":"2023-10-20T10:30:46.162600+0000","mapping_epoch":13,"log_start":"107'9700","ondisk_log_start":"107'9700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":213,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:29:06.831018+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:29:06.831018+0000","last_peered":"2023-10-20T10:29:06.831018+0000","last_clean":"2023-10-20T10:29:06.831018+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:29:06.831018+0000","last_undegraded":"2023-10-20T10:29:06.831018+0000","last_fullsized":"2023-10-20T10:29:06.831018+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"131'9993","reported_seq":15668,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:30:46.162071+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:30:46.162071+0000","last_peered":"2023-10-20T10:30:46.162071+0000","last_clean":"2023-10-20T10:30:46.162071+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:30:46.162071+0000","last_undegraded":"2023-10-20T10:30:46.162071+0000","last_fullsized":"2023-10-20T10:30:46.162071+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":9993,"log_dups_size":0,"ondisk_log_size":9993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"131'516495","reported_seq":1023290,"reported_epoch":131,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:30:47.349028+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:30:47.349028+0000","last_peered":"2023-10-20T10:30:47.349028+0000","last_clean":"2023-10-20T10:30:47.349028+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:30:47.349028+0000","last_undegraded":"2023-10-20T10:30:47.349028+0000","last_fullsized":"2023-10-20T10:30:47.349028+0000","mapping_epoch":16,"log_start":"130'506432","ondisk_log_start":"130'506432","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":4572,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 663s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7209213952,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":509687,"num_read_kb":2008636,"num_write":509484,"num_write_kb":2037936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":307,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:29:06.751943+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:29:06.751943+0000","last_peered":"2023-10-20T10:29:06.751943+0000","last_clean":"2023-10-20T10:29:06.751943+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:29:06.751943+0000","last_undegraded":"2023-10-20T10:29:06.751943+0000","last_fullsized":"2023-10-20T10:29:06.751943+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"131'479659","reported_seq":949727,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:30:47.245479+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:30:47.245479+0000","last_peered":"2023-10-20T10:30:47.245479+0000","last_clean":"2023-10-20T10:30:47.245479+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:30:47.245479+0000","last_undegraded":"2023-10-20T10:30:47.245479+0000","last_fullsized":"2023-10-20T10:30:47.245479+0000","mapping_epoch":16,"log_start":"130'469642","ondisk_log_start":"130'469642","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6685790208,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":472769,"num_read_kb":1861028,"num_write":472915,"num_write_kb":1891660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6722,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:30:46.173597+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:30:46.173597+0000","last_peered":"2023-10-20T10:30:46.173597+0000","last_clean":"2023-10-20T10:30:46.173597+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:30:46.173597+0000","last_undegraded":"2023-10-20T10:30:46.173597+0000","last_fullsized":"2023-10-20T10:30:46.173597+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"131'509456","reported_seq":1008706,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:30:47.334824+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:30:47.334824+0000","last_peered":"2023-10-20T10:30:47.334824+0000","last_clean":"2023-10-20T10:30:47.334824+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:30:47.334824+0000","last_undegraded":"2023-10-20T10:30:47.334824+0000","last_fullsized":"2023-10-20T10:30:47.334824+0000","mapping_epoch":16,"log_start":"130'499436","ondisk_log_start":"130'499436","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7101300736,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":502155,"num_read_kb":1977120,"num_write":502218,"num_write_kb":2008872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6570,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:30:46.162218+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:30:46.162218+0000","last_peered":"2023-10-20T10:30:46.162218+0000","last_clean":"2023-10-20T10:30:46.162218+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:30:46.162218+0000","last_undegraded":"2023-10-20T10:30:46.162218+0000","last_fullsized":"2023-10-20T10:30:46.162218+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"131'504384","reported_seq":1002095,"reported_epoch":131,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:30:47.248524+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:30:47.248524+0000","last_peered":"2023-10-20T10:30:47.248524+0000","last_clean":"2023-10-20T10:30:47.248524+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:30:47.248524+0000","last_undegraded":"2023-10-20T10:30:47.248524+0000","last_fullsized":"2023-10-20T10:30:47.248524+0000","mapping_epoch":16,"log_start":"130'494324","ondisk_log_start":"130'494324","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":917,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 122s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7065145363,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":500714,"num_read_kb":1971720,"num_write":500212,"num_write_kb":2000842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":212,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-20T10:29:06.830969+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:29:06.830969+0000","last_peered":"2023-10-20T10:29:06.830969+0000","last_clean":"2023-10-20T10:29:06.830969+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:29:06.830969+0000","last_undegraded":"2023-10-20T10:29:06.830969+0000","last_fullsized":"2023-10-20T10:29:06.830969+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":306,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:29:38.209891+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:29:38.209891+0000","last_peered":"2023-10-20T10:29:38.209891+0000","last_clean":"2023-10-20T10:29:38.209891+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:29:38.209891+0000","last_undegraded":"2023-10-20T10:29:38.209891+0000","last_fullsized":"2023-10-20T10:29:38.209891+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28061450259,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1985325,"num_read_kb":7818504,"num_write":1984829,"num_write_kb":7939310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16258600960,"data_stored":29641355302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40160,"ondisk_log_size":40160,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":30792,"ondisk_log_size":30792,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739297,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9783568,"kb_used_data":7939528,"kb_used_omap":1747,"kb_used_meta":1842284,"kb_avail":84588272,"statfs":{"total":96636764160,"available":86618390528,"internally_reserved":0,"allocated":8130076672,"data_stored":14820568748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1789111,"internal_metadata":1886499657},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.45700000000000002,"15min":0.47199999999999998},"min":{"1min":0.246,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":0.496,"5min":6.2430000000000003,"15min":8.2449999999999992},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.52000000000000002,"15min":0.48699999999999999},"min":{"1min":0.23599999999999999,"5min":0.20799999999999999,"15min":0.18099999999999999},"max":{"1min":0.73699999999999999,"5min":5.048,"15min":6.8239999999999998},"last":0.28899999999999998}]},{"osd":1,"last update":"Fri Oct 20 10:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.63400000000000001,"15min":0.60699999999999998},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":3.0790000000000002,"5min":6.1509999999999998,"15min":8.282},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.53900000000000003,"15min":0.47499999999999998},"min":{"1min":0.23799999999999999,"5min":0.23400000000000001,"15min":0.214},"max":{"1min":0.72899999999999998,"5min":6.109,"15min":6.109},"last":0.251}]}]},{"osd":1,"up_from":8,"seq":34359739297,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5517996,"kb_used_data":3894244,"kb_used_omap":678,"kb_used_meta":1623065,"kb_avail":88853844,"statfs":{"total":96636764160,"available":90986336256,"internally_reserved":0,"allocated":3987705856,"data_stored":7251860000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":694339,"internal_metadata":1662019517},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,3,4,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.48499999999999999,"15min":0.48299999999999998},"min":{"1min":0.26700000000000002,"5min":0.20000000000000001,"15min":0.17199999999999999},"max":{"1min":1.548,"5min":4.9480000000000004,"15min":4.9480000000000004},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.27300000000000002,"5min":0.46800000000000003,"15min":0.45000000000000001},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.186},"max":{"1min":0.40400000000000003,"5min":5.0049999999999999,"15min":5.8479999999999999},"last":0.24199999999999999}]},{"osd":2,"last update":"Fri Oct 20 10:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.56299999999999994,"15min":0.46100000000000002},"min":{"1min":0.23699999999999999,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":0.60299999999999998,"5min":7.6459999999999999,"15min":7.6459999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.60699999999999998,"15min":0.50700000000000001},"min":{"1min":0.29699999999999999,"5min":0.253,"15min":0.23799999999999999},"max":{"1min":0.627,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.38400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739299,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5860352,"kb_used_data":4048376,"kb_used_omap":700,"kb_used_meta":1811267,"kb_avail":88511488,"statfs":{"total":96636764160,"available":90635763712,"internally_reserved":0,"allocated":4145537024,"data_stored":7570670481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":716869,"internal_metadata":1854738363},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:30:04 2023","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":0.64100000000000001,"15min":0.84599999999999997},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":6.5449999999999999,"5min":6.5449999999999999,"15min":81.197000000000003},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.502,"15min":0.78000000000000003},"min":{"1min":0.20699999999999999,"5min":0.19800000000000001,"15min":0.184},"max":{"1min":6.024,"5min":6.2229999999999999,"15min":80.334999999999994},"last":0.33700000000000002}]},{"osd":2,"last update":"Fri Oct 20 10:30:04 2023","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":0.60899999999999999,"15min":0.52900000000000003},"min":{"1min":0.252,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":6.3029999999999999,"5min":6.3029999999999999,"15min":11.176},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.47899999999999998,"15min":0.501},"min":{"1min":0.24299999999999999,"5min":0.21099999999999999,"15min":0.20000000000000001},"max":{"1min":4.0220000000000002,"5min":4.4260000000000002,"15min":11.093},"last":0.28699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4143931392,"data_stored":7570034688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3985928192,"data_stored":7251202067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8128741376,"data_stored":14820118547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:30:50.447 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:30:50.447 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:30:50.448 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:31:10.450 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:31:11.018 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:31:11.019 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:31:11.049 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2459,"stamp":"2023-10-20T10:31:09.328955+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28091920899,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2031282,"num_read_kb":7880313,"num_write":1993121,"num_write_kb":7970735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71230,"ondisk_log_size":71230,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21431244,"kb_used_data":15938676,"kb_used_omap":3125,"kb_used_meta":5489418,"kb_avail":261684276,"statfs":{"total":289910292480,"available":267964698624,"internally_reserved":0,"allocated":16321204224,"data_stored":29700972375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3200319,"internal_metadata":5621164737},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,5,11,10,23,13,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15007744,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3730,"num_read_kb":14896,"num_write":3644,"num_write_kb":14576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.116811"},"pg_stats":[{"pgid":"2.7","version":"132'10068","reported_seq":14981,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:06.638119+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:31:06.638119+0000","last_peered":"2023-10-20T10:31:06.638119+0000","last_clean":"2023-10-20T10:31:06.638119+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:31:06.638119+0000","last_undegraded":"2023-10-20T10:31:06.638119+0000","last_fullsized":"2023-10-20T10:31:06.638119+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"132'19799","reported_seq":30603,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:06.638821+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:31:06.638821+0000","last_peered":"2023-10-20T10:31:06.638821+0000","last_clean":"2023-10-20T10:31:06.638821+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:31:06.638821+0000","last_undegraded":"2023-10-20T10:31:06.638821+0000","last_fullsized":"2023-10-20T10:31:06.638821+0000","mapping_epoch":13,"log_start":"107'9700","ondisk_log_start":"107'9700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":214,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:29:38.201399+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:29:38.201399+0000","last_peered":"2023-10-20T10:29:38.201399+0000","last_clean":"2023-10-20T10:29:38.201399+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:29:38.201399+0000","last_undegraded":"2023-10-20T10:29:38.201399+0000","last_fullsized":"2023-10-20T10:29:38.201399+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"132'10029","reported_seq":15717,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:06.638188+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:31:06.638188+0000","last_peered":"2023-10-20T10:31:06.638188+0000","last_clean":"2023-10-20T10:31:06.638188+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:31:06.638188+0000","last_undegraded":"2023-10-20T10:31:06.638188+0000","last_fullsized":"2023-10-20T10:31:06.638188+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"132'518483","reported_seq":1027160,"reported_epoch":132,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:31:07.452161+0000","last_change":"2023-10-20T10:19:43.907833+0000","last_active":"2023-10-20T10:31:07.452161+0000","last_peered":"2023-10-20T10:31:07.452161+0000","last_clean":"2023-10-20T10:31:07.452161+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:31:07.452161+0000","last_undegraded":"2023-10-20T10:31:07.452161+0000","last_fullsized":"2023-10-20T10:31:07.452161+0000","mapping_epoch":16,"log_start":"130'508432","ondisk_log_start":"130'508432","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":4744,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 683s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7216812032,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":511554,"num_read_kb":2016096,"num_write":511340,"num_write_kb":2045360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":308,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:29:38.202614+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:29:38.202614+0000","last_peered":"2023-10-20T10:29:38.202614+0000","last_clean":"2023-10-20T10:29:38.202614+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:29:38.202614+0000","last_undegraded":"2023-10-20T10:29:38.202614+0000","last_fullsized":"2023-10-20T10:29:38.202614+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"132'481385","reported_seq":953207,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:07.332353+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:31:07.332353+0000","last_peered":"2023-10-20T10:31:07.332353+0000","last_clean":"2023-10-20T10:31:07.332353+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:31:07.332353+0000","last_undegraded":"2023-10-20T10:31:07.332353+0000","last_fullsized":"2023-10-20T10:31:07.332353+0000","mapping_epoch":16,"log_start":"130'471342","ondisk_log_start":"130'471342","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6692884480,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":474521,"num_read_kb":1868016,"num_write":474641,"num_write_kb":1898564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6740,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:06.638698+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:31:06.638698+0000","last_peered":"2023-10-20T10:31:06.638698+0000","last_clean":"2023-10-20T10:31:06.638698+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:31:06.638698+0000","last_undegraded":"2023-10-20T10:31:06.638698+0000","last_fullsized":"2023-10-20T10:31:06.638698+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"132'511322","reported_seq":1012415,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:07.442159+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:31:07.442159+0000","last_peered":"2023-10-20T10:31:07.442159+0000","last_clean":"2023-10-20T10:31:07.442159+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:31:07.442159+0000","last_undegraded":"2023-10-20T10:31:07.442159+0000","last_fullsized":"2023-10-20T10:31:07.442159+0000","mapping_epoch":16,"log_start":"130'501236","ondisk_log_start":"130'501236","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7108960256,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":503997,"num_read_kb":1984480,"num_write":504084,"num_write_kb":2016336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6587,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:06.639930+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:31:06.639930+0000","last_peered":"2023-10-20T10:31:06.639930+0000","last_clean":"2023-10-20T10:31:06.639930+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:31:06.639930+0000","last_undegraded":"2023-10-20T10:31:06.639930+0000","last_fullsized":"2023-10-20T10:31:06.639930+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"132'506339","reported_seq":1005895,"reported_epoch":132,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:31:07.337916+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:31:07.337916+0000","last_peered":"2023-10-20T10:31:07.337916+0000","last_clean":"2023-10-20T10:31:07.337916+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:31:07.337916+0000","last_undegraded":"2023-10-20T10:31:07.337916+0000","last_fullsized":"2023-10-20T10:31:07.337916+0000","mapping_epoch":16,"log_start":"130'496324","ondisk_log_start":"130'496324","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":1079,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 142s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7072669715,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":502539,"num_read_kb":1979000,"num_write":502042,"num_write_kb":2008162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":213,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:29:38.201376+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:29:38.201376+0000","last_peered":"2023-10-20T10:29:38.201376+0000","last_clean":"2023-10-20T10:29:38.201376+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:29:38.201376+0000","last_undegraded":"2023-10-20T10:29:38.201376+0000","last_fullsized":"2023-10-20T10:29:38.201376+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":307,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:30:02.079100+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:30:02.079100+0000","last_peered":"2023-10-20T10:30:02.079100+0000","last_clean":"2023-10-20T10:30:02.079100+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:30:02.079100+0000","last_undegraded":"2023-10-20T10:30:02.079100+0000","last_fullsized":"2023-10-20T10:30:02.079100+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28091326483,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":1992611,"num_read_kb":7847592,"num_write":1992107,"num_write_kb":7968422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16318218240,"data_stored":29700972582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40195,"ondisk_log_size":40195,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":30936,"ondisk_log_size":30936,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739301,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9920016,"kb_used_data":7967240,"kb_used_omap":1747,"kb_used_meta":1951020,"kb_avail":84451824,"statfs":{"total":96636764160,"available":86478667776,"internally_reserved":0,"allocated":8158453760,"data_stored":14848941994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1789111,"internal_metadata":1997845321},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.45700000000000002,"15min":0.47199999999999998},"min":{"1min":0.246,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":0.496,"5min":6.2430000000000003,"15min":8.2449999999999992},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.52000000000000002,"15min":0.48699999999999999},"min":{"1min":0.23599999999999999,"5min":0.20799999999999999,"15min":0.18099999999999999},"max":{"1min":0.73699999999999999,"5min":5.048,"15min":6.8239999999999998},"last":0.32500000000000001}]},{"osd":1,"last update":"Fri Oct 20 10:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.63400000000000001,"15min":0.60699999999999998},"min":{"1min":0.20699999999999999,"5min":0.186,"15min":0.186},"max":{"1min":3.0790000000000002,"5min":6.1509999999999998,"15min":8.282},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.53900000000000003,"15min":0.47499999999999998},"min":{"1min":0.23799999999999999,"5min":0.23400000000000001,"15min":0.214},"max":{"1min":0.72899999999999998,"5min":6.109,"15min":6.109},"last":0.27800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739301,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5564000,"kb_used_data":3908312,"kb_used_omap":678,"kb_used_meta":1655001,"kb_avail":88807840,"statfs":{"total":96636764160,"available":90939228160,"internally_reserved":0,"allocated":4002111488,"data_stored":7266261790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":694339,"internal_metadata":1694721981},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,2,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.48499999999999999,"15min":0.48299999999999998},"min":{"1min":0.26700000000000002,"5min":0.20000000000000001,"15min":0.17199999999999999},"max":{"1min":1.548,"5min":4.9480000000000004,"15min":4.9480000000000004},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.27300000000000002,"5min":0.46800000000000003,"15min":0.45000000000000001},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.186},"max":{"1min":0.40400000000000003,"5min":5.0049999999999999,"15min":5.8479999999999999},"last":0.25}]},{"osd":2,"last update":"Fri Oct 20 10:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.56299999999999994,"15min":0.46100000000000002},"min":{"1min":0.23699999999999999,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":0.60299999999999998,"5min":7.6459999999999999,"15min":7.6459999999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.60699999999999998,"15min":0.50700000000000001},"min":{"1min":0.29699999999999999,"5min":0.253,"15min":0.23799999999999999},"max":{"1min":0.627,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.372}]}]},{"osd":0,"up_from":8,"seq":34359739303,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5947228,"kb_used_data":4063124,"kb_used_omap":700,"kb_used_meta":1883395,"kb_avail":88424612,"statfs":{"total":96636764160,"available":90546802688,"internally_reserved":0,"allocated":4160638976,"data_stored":7585768591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":716869,"internal_metadata":1928597435},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,5,13,9,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:30:04 2023","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":0.64100000000000001,"15min":0.84599999999999997},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":6.5449999999999999,"5min":6.5449999999999999,"15min":81.197000000000003},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.502,"15min":0.78000000000000003},"min":{"1min":0.20699999999999999,"5min":0.19800000000000001,"15min":0.184},"max":{"1min":6.024,"5min":6.2229999999999999,"15min":80.334999999999994},"last":0.41399999999999998}]},{"osd":2,"last update":"Fri Oct 20 10:30:04 2023","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":0.60899999999999999,"15min":0.52900000000000003},"min":{"1min":0.252,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":6.3029999999999999,"5min":6.3029999999999999,"15min":11.176},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.47899999999999998,"15min":0.501},"min":{"1min":0.24299999999999999,"5min":0.21099999999999999,"15min":0.20000000000000001},"max":{"1min":4.0220000000000002,"5min":4.4260000000000002,"15min":11.093},"last":0.32700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4159176704,"data_stored":7585280000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4000493568,"data_stored":7265767443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8158547968,"data_stored":14849925139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:31:11.051 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:31:11.051 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:31:11.052 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:31:23.176 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:31:23.176+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:31:23.178 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:31:23.176+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:31:31.053 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:31:31.476 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:31:31.477 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:31:31.524 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2470,"stamp":"2023-10-20T10:31:31.335737+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28123677187,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2039060,"num_read_kb":7911361,"num_write":2000836,"num_write_kb":8001595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71221,"ondisk_log_size":71221,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21634528,"kb_used_data":16001736,"kb_used_omap":3125,"kb_used_meta":5629642,"kb_avail":261480992,"statfs":{"total":289910292480,"available":267756535808,"internally_reserved":0,"allocated":16385777664,"data_stored":29765545815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3200319,"internal_metadata":5764754113},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,8,6,7,7,17,17],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16035840,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3906,"num_read_kb":15596,"num_write":3883,"num_write_kb":15532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005077"},"pg_stats":[{"pgid":"2.7","version":"132'10117","reported_seq":15042,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:27.389230+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:31:27.389230+0000","last_peered":"2023-10-20T10:31:27.389230+0000","last_clean":"2023-10-20T10:31:27.389230+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:31:27.389230+0000","last_undegraded":"2023-10-20T10:31:27.389230+0000","last_fullsized":"2023-10-20T10:31:27.389230+0000","mapping_epoch":13,"log_start":"40'100","ondisk_log_start":"40'100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"132'19896","reported_seq":30724,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:27.403171+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:31:27.403171+0000","last_peered":"2023-10-20T10:31:27.403171+0000","last_clean":"2023-10-20T10:31:27.403171+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:31:27.403171+0000","last_undegraded":"2023-10-20T10:31:27.403171+0000","last_fullsized":"2023-10-20T10:31:27.403171+0000","mapping_epoch":13,"log_start":"107'9800","ondisk_log_start":"107'9800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":215,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:30:58.223678+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:30:58.223678+0000","last_peered":"2023-10-20T10:30:58.223678+0000","last_clean":"2023-10-20T10:30:58.223678+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:30:58.223678+0000","last_undegraded":"2023-10-20T10:30:58.223678+0000","last_fullsized":"2023-10-20T10:30:58.223678+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"132'10078","reported_seq":15779,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:27.393523+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:31:27.393523+0000","last_peered":"2023-10-20T10:31:27.393523+0000","last_clean":"2023-10-20T10:31:27.393523+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:31:27.393523+0000","last_undegraded":"2023-10-20T10:31:27.393523+0000","last_fullsized":"2023-10-20T10:31:27.393523+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"132'520553","reported_seq":1031236,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:27.479160+0000","last_change":"2023-10-20T10:31:19.489466+0000","last_active":"2023-10-20T10:31:27.479160+0000","last_peered":"2023-10-20T10:31:27.479160+0000","last_clean":"2023-10-20T10:31:27.479160+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:31:27.479160+0000","last_undegraded":"2023-10-20T10:31:27.479160+0000","last_fullsized":"2023-10-20T10:31:27.479160+0000","mapping_epoch":16,"log_start":"131'510532","ondisk_log_start":"131'510532","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'519713","last_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_deep_scrub":"132'519713","last_deep_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_clean_scrub_stamp":"2023-10-20T10:31:19.489275+0000","objects_scrubbed":4836,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":696,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:01:22.001899+0000","scrub_duration":695.57415461999994,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7225044992,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":513545,"num_read_kb":2024032,"num_write":513341,"num_write_kb":2053364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":309,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:30:58.213622+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:30:58.213622+0000","last_peered":"2023-10-20T10:30:58.213622+0000","last_clean":"2023-10-20T10:30:58.213622+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:30:58.213622+0000","last_undegraded":"2023-10-20T10:30:58.213622+0000","last_fullsized":"2023-10-20T10:30:58.213622+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"132'483275","reported_seq":956999,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:27.425672+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:31:27.425672+0000","last_peered":"2023-10-20T10:31:27.425672+0000","last_clean":"2023-10-20T10:31:27.425672+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:31:27.425672+0000","last_undegraded":"2023-10-20T10:31:27.425672+0000","last_fullsized":"2023-10-20T10:31:27.425672+0000","mapping_epoch":16,"log_start":"130'473242","ondisk_log_start":"130'473242","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6700703744,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":476422,"num_read_kb":1875600,"num_write":476531,"num_write_kb":1906124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6752,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:22.003988+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:31:22.003988+0000","last_peered":"2023-10-20T10:31:22.003988+0000","last_clean":"2023-10-20T10:31:22.003988+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:31:22.003988+0000","last_undegraded":"2023-10-20T10:31:22.003988+0000","last_fullsized":"2023-10-20T10:31:22.003988+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"132'513220","reported_seq":1016291,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:27.482399+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:31:27.482399+0000","last_peered":"2023-10-20T10:31:27.482399+0000","last_clean":"2023-10-20T10:31:27.482399+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:31:27.482399+0000","last_undegraded":"2023-10-20T10:31:27.482399+0000","last_fullsized":"2023-10-20T10:31:27.482399+0000","mapping_epoch":16,"log_start":"131'503136","ondisk_log_start":"131'503136","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7116763136,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":505977,"num_read_kb":1992388,"num_write":505982,"num_write_kb":2023928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6599,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:21.998288+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:31:21.998288+0000","last_peered":"2023-10-20T10:31:21.998288+0000","last_clean":"2023-10-20T10:31:21.998288+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:31:21.998288+0000","last_undegraded":"2023-10-20T10:31:21.998288+0000","last_fullsized":"2023-10-20T10:31:21.998288+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"132'508377","reported_seq":1009848,"reported_epoch":132,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:31:27.436301+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:31:27.436301+0000","last_peered":"2023-10-20T10:31:27.436301+0000","last_clean":"2023-10-20T10:31:27.436301+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:31:27.436301+0000","last_undegraded":"2023-10-20T10:31:27.436301+0000","last_fullsized":"2023-10-20T10:31:27.436301+0000","mapping_epoch":16,"log_start":"131'498324","ondisk_log_start":"131'498324","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":1221,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 162s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7080570899,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":504445,"num_read_kb":1986620,"num_write":503968,"num_write_kb":2015866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":214,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:30:58.223459+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:30:58.223459+0000","last_peered":"2023-10-20T10:30:58.223459+0000","last_clean":"2023-10-20T10:30:58.223459+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:30:58.223459+0000","last_undegraded":"2023-10-20T10:30:58.223459+0000","last_fullsized":"2023-10-20T10:30:58.223459+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":307,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:30:02.079100+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:30:02.079100+0000","last_peered":"2023-10-20T10:30:02.079100+0000","last_clean":"2023-10-20T10:30:02.079100+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:30:02.079100+0000","last_undegraded":"2023-10-20T10:30:02.079100+0000","last_fullsized":"2023-10-20T10:30:02.079100+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28123082771,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":2000389,"num_read_kb":7878640,"num_write":1999822,"num_write_kb":7999282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16381423616,"data_stored":29764177958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40191,"ondisk_log_size":40191,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":30931,"ondisk_log_size":30931,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739305,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10038396,"kb_used_data":7999540,"kb_used_omap":1747,"kb_used_meta":2037100,"kb_avail":84333444,"statfs":{"total":96636764160,"available":86357446656,"internally_reserved":0,"allocated":8191528960,"data_stored":14882017194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1789111,"internal_metadata":2085991241},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:31:11 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.45600000000000002,"15min":0.48799999999999999},"min":{"1min":0.189,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":4.3650000000000002,"5min":6.2430000000000003,"15min":8.2449999999999992},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.50800000000000001,"15min":0.498},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.18099999999999999},"max":{"1min":4.3840000000000003,"5min":5.048,"15min":6.8239999999999998},"last":0.33000000000000002}]},{"osd":1,"last update":"Fri Oct 20 10:31:11 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.622,"15min":0.60299999999999998},"min":{"1min":0.21099999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.60199999999999998,"5min":6.1509999999999998,"15min":8.282},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.54200000000000004,"15min":0.47799999999999998},"min":{"1min":0.251,"5min":0.23400000000000001,"15min":0.214},"max":{"1min":1.55,"5min":6.109,"15min":6.109},"last":0.29499999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739305,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5597172,"kb_used_data":3923436,"kb_used_omap":678,"kb_used_meta":1673049,"kb_avail":88774668,"statfs":{"total":96636764160,"available":90905260032,"internally_reserved":0,"allocated":4017598464,"data_stored":7281748766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":694339,"internal_metadata":1713203133},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,1,16,17],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:31:10 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.48399999999999999,"15min":0.47899999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.17199999999999999},"max":{"1min":0.51600000000000001,"5min":4.9480000000000004,"15min":4.9480000000000004},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.48599999999999999,"15min":0.45600000000000002},"min":{"1min":0.21099999999999999,"5min":0.19400000000000001,"15min":0.186},"max":{"1min":1.4099999999999999,"5min":5.0049999999999999,"15min":5.8479999999999999},"last":0.23499999999999999}]},{"osd":2,"last update":"Fri Oct 20 10:31:10 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.51800000000000002,"15min":0.46500000000000002},"min":{"1min":0.22700000000000001,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":1.3140000000000001,"5min":7.6459999999999999,"15min":7.6459999999999999},"last":0.251},{"interface":"front","average":{"1min":0.435,"5min":0.56100000000000005,"15min":0.50900000000000001},"min":{"1min":0.28199999999999997,"5min":0.253,"15min":0.23799999999999999},"max":{"1min":1.3420000000000001,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.27600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739307,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5998960,"kb_used_data":4078760,"kb_used_omap":700,"kb_used_meta":1919491,"kb_avail":88372880,"statfs":{"total":96636764160,"available":90493829120,"internally_reserved":0,"allocated":4176650240,"data_stored":7601779855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":716869,"internal_metadata":1965559739},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:31:08 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.63900000000000001,"15min":0.58799999999999997},"min":{"1min":0.182,"5min":0.182,"15min":0.17599999999999999},"max":{"1min":0.504,"5min":6.5449999999999999,"15min":9.266},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.502,"15min":0.51000000000000001},"min":{"1min":0.223,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.55600000000000005,"5min":6.2229999999999999,"15min":7.2690000000000001},"last":0.26400000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:31:08 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.62,"15min":0.53400000000000003},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":0.96199999999999997,"5min":6.3029999999999999,"15min":11.176},"last":0.315},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.49099999999999999,"15min":0.503},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20000000000000001},"max":{"1min":0.94699999999999995,"5min":4.4260000000000002,"15min":11.093},"last":0.29499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4175151104,"data_stored":7601254400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4016119808,"data_stored":7281393683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8190152704,"data_stored":14881529875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:31:31.526 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:31:31.527 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:31:51.528 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:31:51.986 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:31:51.987 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:31:52.019 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2480,"stamp":"2023-10-20T10:31:51.338900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28155142659,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2046849,"num_read_kb":7942425,"num_write":2008471,"num_write_kb":8032135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71284,"ondisk_log_size":71284,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21893356,"kb_used_data":16062100,"kb_used_omap":3125,"kb_used_meta":5828106,"kb_avail":261222164,"statfs":{"total":289910292480,"available":267491495936,"internally_reserved":0,"allocated":16447590400,"data_stored":29827358551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3200319,"internal_metadata":5967981249},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,9,7,10,10,10,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15958016,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3866,"num_read_kb":15416,"num_write":3883,"num_write_kb":15532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001861"},"pg_stats":[{"pgid":"2.7","version":"132'10165","reported_seq":15099,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:47.310593+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:31:47.310593+0000","last_peered":"2023-10-20T10:31:47.310593+0000","last_clean":"2023-10-20T10:31:47.310593+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:31:47.310593+0000","last_undegraded":"2023-10-20T10:31:47.310593+0000","last_fullsized":"2023-10-20T10:31:47.310593+0000","mapping_epoch":13,"log_start":"40'100","ondisk_log_start":"40'100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"132'19992","reported_seq":30838,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:47.337010+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:31:47.337010+0000","last_peered":"2023-10-20T10:31:47.337010+0000","last_clean":"2023-10-20T10:31:47.337010+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:31:47.337010+0000","last_undegraded":"2023-10-20T10:31:47.337010+0000","last_fullsized":"2023-10-20T10:31:47.337010+0000","mapping_epoch":13,"log_start":"109'9900","ondisk_log_start":"109'9900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":215,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:30:58.223678+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:30:58.223678+0000","last_peered":"2023-10-20T10:30:58.223678+0000","last_clean":"2023-10-20T10:30:58.223678+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:30:58.223678+0000","last_undegraded":"2023-10-20T10:30:58.223678+0000","last_fullsized":"2023-10-20T10:30:58.223678+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"132'10126","reported_seq":15836,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:47.312100+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:31:47.312100+0000","last_peered":"2023-10-20T10:31:47.312100+0000","last_clean":"2023-10-20T10:31:47.312100+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:31:47.312100+0000","last_undegraded":"2023-10-20T10:31:47.312100+0000","last_fullsized":"2023-10-20T10:31:47.312100+0000","mapping_epoch":13,"log_start":"40'100","ondisk_log_start":"40'100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"132'522520","reported_seq":1035215,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:47.525084+0000","last_change":"2023-10-20T10:31:19.489466+0000","last_active":"2023-10-20T10:31:47.525084+0000","last_peered":"2023-10-20T10:31:47.525084+0000","last_clean":"2023-10-20T10:31:47.525084+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:31:47.525084+0000","last_undegraded":"2023-10-20T10:31:47.525084+0000","last_fullsized":"2023-10-20T10:31:47.525084+0000","mapping_epoch":16,"log_start":"131'512432","ondisk_log_start":"131'512432","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'519713","last_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_deep_scrub":"132'519713","last_deep_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_clean_scrub_stamp":"2023-10-20T10:31:19.489275+0000","objects_scrubbed":4836,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":696,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:01:22.001899+0000","scrub_duration":695.57415461999994,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7233126400,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":515558,"num_read_kb":2032056,"num_write":515308,"num_write_kb":2061232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":309,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:30:58.213622+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:30:58.213622+0000","last_peered":"2023-10-20T10:30:58.213622+0000","last_clean":"2023-10-20T10:30:58.213622+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:30:58.213622+0000","last_undegraded":"2023-10-20T10:30:58.213622+0000","last_fullsized":"2023-10-20T10:30:58.213622+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"132'485096","reported_seq":960744,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:47.573170+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:31:47.573170+0000","last_peered":"2023-10-20T10:31:47.573170+0000","last_clean":"2023-10-20T10:31:47.573170+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:31:47.573170+0000","last_undegraded":"2023-10-20T10:31:47.573170+0000","last_fullsized":"2023-10-20T10:31:47.573170+0000","mapping_epoch":16,"log_start":"131'475042","ondisk_log_start":"131'475042","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6708232192,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":478346,"num_read_kb":1883264,"num_write":478352,"num_write_kb":1913408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6768,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:42.482722+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:31:42.482722+0000","last_peered":"2023-10-20T10:31:42.482722+0000","last_clean":"2023-10-20T10:31:42.482722+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:31:42.482722+0000","last_undegraded":"2023-10-20T10:31:42.482722+0000","last_fullsized":"2023-10-20T10:31:42.482722+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"132'515156","reported_seq":1020106,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:47.540849+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:31:47.540849+0000","last_peered":"2023-10-20T10:31:47.540849+0000","last_clean":"2023-10-20T10:31:47.540849+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:31:47.540849+0000","last_undegraded":"2023-10-20T10:31:47.540849+0000","last_fullsized":"2023-10-20T10:31:47.540849+0000","mapping_epoch":16,"log_start":"131'505136","ondisk_log_start":"131'505136","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7124733952,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":507856,"num_read_kb":1999884,"num_write":507918,"num_write_kb":2031672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6615,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:42.482497+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:31:42.482497+0000","last_peered":"2023-10-20T10:31:42.482497+0000","last_clean":"2023-10-20T10:31:42.482497+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:31:42.482497+0000","last_undegraded":"2023-10-20T10:31:42.482497+0000","last_fullsized":"2023-10-20T10:31:42.482497+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"132'510424","reported_seq":1013883,"reported_epoch":132,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:31:47.583248+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:31:47.583248+0000","last_peered":"2023-10-20T10:31:47.583248+0000","last_clean":"2023-10-20T10:31:47.583248+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:31:47.583248+0000","last_undegraded":"2023-10-20T10:31:47.583248+0000","last_fullsized":"2023-10-20T10:31:47.583248+0000","mapping_epoch":16,"log_start":"131'500324","ondisk_log_start":"131'500324","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":1399,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 182s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7088455699,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":506418,"num_read_kb":1994500,"num_write":505879,"num_write_kb":2023510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":214,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:30:58.223459+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:30:58.223459+0000","last_peered":"2023-10-20T10:30:58.223459+0000","last_clean":"2023-10-20T10:30:58.223459+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:30:58.223459+0000","last_undegraded":"2023-10-20T10:30:58.223459+0000","last_fullsized":"2023-10-20T10:30:58.223459+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":307,"reported_epoch":131,"state":"active+clean","last_fresh":"2023-10-20T10:30:02.079100+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:30:02.079100+0000","last_peered":"2023-10-20T10:30:02.079100+0000","last_clean":"2023-10-20T10:30:02.079100+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:30:02.079100+0000","last_undegraded":"2023-10-20T10:30:02.079100+0000","last_fullsized":"2023-10-20T10:30:02.079100+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28154548243,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":2008178,"num_read_kb":7909704,"num_write":2007457,"num_write_kb":8029822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16443899904,"data_stored":29826654246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40262,"ondisk_log_size":40262,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":30923,"ondisk_log_size":30923,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739309,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10153568,"kb_used_data":8028632,"kb_used_omap":1747,"kb_used_meta":2123180,"kb_avail":84218272,"statfs":{"total":96636764160,"available":86239510528,"internally_reserved":0,"allocated":8221319168,"data_stored":14911807402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1789111,"internal_metadata":2174137161},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:31:11 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.45600000000000002,"15min":0.48799999999999999},"min":{"1min":0.189,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":4.3650000000000002,"5min":6.2430000000000003,"15min":8.2449999999999992},"last":0.246},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.50800000000000001,"15min":0.498},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.18099999999999999},"max":{"1min":4.3840000000000003,"5min":5.048,"15min":6.8239999999999998},"last":0.32200000000000001}]},{"osd":1,"last update":"Fri Oct 20 10:31:11 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.622,"15min":0.60299999999999998},"min":{"1min":0.21099999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.60199999999999998,"5min":6.1509999999999998,"15min":8.282},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.54200000000000004,"15min":0.47799999999999998},"min":{"1min":0.251,"5min":0.23400000000000001,"15min":0.214},"max":{"1min":1.55,"5min":6.109,"15min":6.109},"last":0.28799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739309,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5671156,"kb_used_data":3939116,"kb_used_omap":678,"kb_used_meta":1731353,"kb_avail":88700684,"statfs":{"total":96636764160,"available":90829500416,"internally_reserved":0,"allocated":4033654784,"data_stored":7297805086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":694339,"internal_metadata":1772906429},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,2,4,4,10,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:31:10 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.48399999999999999,"15min":0.47899999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.17199999999999999},"max":{"1min":0.51600000000000001,"5min":4.9480000000000004,"15min":4.9480000000000004},"last":0.221},{"interface":"front","average":{"1min":0.372,"5min":0.48599999999999999,"15min":0.45600000000000002},"min":{"1min":0.21099999999999999,"5min":0.19400000000000001,"15min":0.186},"max":{"1min":1.4099999999999999,"5min":5.0049999999999999,"15min":5.8479999999999999},"last":0.29299999999999998}]},{"osd":2,"last update":"Fri Oct 20 10:31:10 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.51800000000000002,"15min":0.46500000000000002},"min":{"1min":0.22700000000000001,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":1.3140000000000001,"5min":7.6459999999999999,"15min":7.6459999999999999},"last":0.253},{"interface":"front","average":{"1min":0.435,"5min":0.56100000000000005,"15min":0.50900000000000001},"min":{"1min":0.28199999999999997,"5min":0.253,"15min":0.23799999999999999},"max":{"1min":1.3420000000000001,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.27900000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739311,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6068632,"kb_used_data":4094352,"kb_used_omap":700,"kb_used_meta":1973571,"kb_avail":88303208,"statfs":{"total":96636764160,"available":90422484992,"internally_reserved":0,"allocated":4192616448,"data_stored":7617746063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":716869,"internal_metadata":2020937659},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:31:08 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.63900000000000001,"15min":0.58799999999999997},"min":{"1min":0.182,"5min":0.182,"15min":0.17599999999999999},"max":{"1min":0.504,"5min":6.5449999999999999,"15min":9.266},"last":0.625},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.502,"15min":0.51000000000000001},"min":{"1min":0.223,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.55600000000000005,"5min":6.2229999999999999,"15min":7.2690000000000001},"last":0.59199999999999997}]},{"osd":2,"last update":"Fri Oct 20 10:31:08 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.62,"15min":0.53400000000000003},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":0.96199999999999997,"5min":6.3029999999999999,"15min":11.176},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.49099999999999999,"15min":0.503},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20000000000000001},"max":{"1min":0.94699999999999995,"5min":4.4260000000000002,"15min":11.093},"last":0.22800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4191137792,"data_stored":7617241088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4031410176,"data_stored":7296684051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8221351936,"data_stored":14912729107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:31:52.021 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:31:52.022 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:32:05.999 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:32:05.999+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:32:09.238 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:32:09.235+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:32:09.239 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:32:09.235+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:32:09.250 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:32:09.243+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:32:12.023 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:32:12.544 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:32:12.546 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:32:12.603 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2491,"stamp":"2023-10-20T10:32:11.343738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28186374659,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2054803,"num_read_kb":7974117,"num_write":2016055,"num_write_kb":8062471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71160,"ondisk_log_size":71160,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21731464,"kb_used_data":16123056,"kb_used_omap":3125,"kb_used_meta":5605258,"kb_avail":261384056,"statfs":{"total":289910292480,"available":267657273344,"internally_reserved":0,"allocated":16510009344,"data_stored":29889777495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3200319,"internal_metadata":5739784897},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,6,3,9,6,11,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15781888,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3914,"num_read_kb":15600,"num_write":3826,"num_write_kb":15304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001568"},"pg_stats":[{"pgid":"2.7","version":"132'10214","reported_seq":15161,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:32:06.024391+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:32:06.024391+0000","last_peered":"2023-10-20T10:32:06.024391+0000","last_clean":"2023-10-20T10:32:06.024391+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:32:06.024391+0000","last_undegraded":"2023-10-20T10:32:06.024391+0000","last_fullsized":"2023-10-20T10:32:06.024391+0000","mapping_epoch":13,"log_start":"42'200","ondisk_log_start":"42'200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"132'20089","reported_seq":30960,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:32:06.038995+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:32:06.038995+0000","last_peered":"2023-10-20T10:32:06.038995+0000","last_clean":"2023-10-20T10:32:06.038995+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:32:06.038995+0000","last_undegraded":"2023-10-20T10:32:06.038995+0000","last_fullsized":"2023-10-20T10:32:06.038995+0000","mapping_epoch":13,"log_start":"109'10000","ondisk_log_start":"109'10000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":215,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:30:58.223678+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:30:58.223678+0000","last_peered":"2023-10-20T10:30:58.223678+0000","last_clean":"2023-10-20T10:30:58.223678+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:30:58.223678+0000","last_undegraded":"2023-10-20T10:30:58.223678+0000","last_fullsized":"2023-10-20T10:30:58.223678+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"132'10175","reported_seq":15897,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:32:06.023262+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:32:06.023262+0000","last_peered":"2023-10-20T10:32:06.023262+0000","last_clean":"2023-10-20T10:32:06.023262+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:32:06.023262+0000","last_undegraded":"2023-10-20T10:32:06.023262+0000","last_fullsized":"2023-10-20T10:32:06.023262+0000","mapping_epoch":13,"log_start":"40'100","ondisk_log_start":"40'100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"132'524453","reported_seq":1039179,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:32:07.659786+0000","last_change":"2023-10-20T10:31:19.489466+0000","last_active":"2023-10-20T10:32:07.659786+0000","last_peered":"2023-10-20T10:32:07.659786+0000","last_clean":"2023-10-20T10:32:07.659786+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:32:07.659786+0000","last_undegraded":"2023-10-20T10:32:07.659786+0000","last_fullsized":"2023-10-20T10:32:07.659786+0000","mapping_epoch":16,"log_start":"131'514432","ondisk_log_start":"131'514432","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'519713","last_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_deep_scrub":"132'519713","last_deep_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_clean_scrub_stamp":"2023-10-20T10:31:19.489275+0000","objects_scrubbed":4836,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":696,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:01:22.001899+0000","scrub_duration":695.57415461999994,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7241072640,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":517588,"num_read_kb":2040164,"num_write":517241,"num_write_kb":2068964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":309,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:30:58.213622+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:30:58.213622+0000","last_peered":"2023-10-20T10:30:58.213622+0000","last_clean":"2023-10-20T10:30:58.213622+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:30:58.213622+0000","last_undegraded":"2023-10-20T10:30:58.213622+0000","last_fullsized":"2023-10-20T10:30:58.213622+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"132'486866","reported_seq":964431,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:32:07.666124+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:32:07.666124+0000","last_peered":"2023-10-20T10:32:07.666124+0000","last_clean":"2023-10-20T10:32:07.666124+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:32:07.666124+0000","last_undegraded":"2023-10-20T10:32:07.666124+0000","last_fullsized":"2023-10-20T10:32:07.666124+0000","mapping_epoch":16,"log_start":"131'476842","ondisk_log_start":"131'476842","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6715527168,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":480264,"num_read_kb":1890900,"num_write":480122,"num_write_kb":1920488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6784,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:32:02.959552+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:32:02.959552+0000","last_peered":"2023-10-20T10:32:02.959552+0000","last_clean":"2023-10-20T10:32:02.959552+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:32:02.959552+0000","last_undegraded":"2023-10-20T10:32:02.959552+0000","last_fullsized":"2023-10-20T10:32:02.959552+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"132'517044","reported_seq":1024015,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:32:07.650979+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:32:07.650979+0000","last_peered":"2023-10-20T10:32:07.650979+0000","last_clean":"2023-10-20T10:32:07.650979+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:32:07.650979+0000","last_undegraded":"2023-10-20T10:32:07.650979+0000","last_fullsized":"2023-10-20T10:32:07.650979+0000","mapping_epoch":16,"log_start":"131'507036","ondisk_log_start":"131'507036","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7132545024,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":509878,"num_read_kb":2007936,"num_write":509806,"num_write_kb":2039224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6632,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:32:02.958876+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:32:02.958876+0000","last_peered":"2023-10-20T10:32:02.958876+0000","last_clean":"2023-10-20T10:32:02.958876+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:32:02.958876+0000","last_undegraded":"2023-10-20T10:32:02.958876+0000","last_fullsized":"2023-10-20T10:32:02.958876+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"132'512514","reported_seq":1017969,"reported_epoch":132,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:32:07.692953+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:32:07.692953+0000","last_peered":"2023-10-20T10:32:07.692953+0000","last_clean":"2023-10-20T10:32:07.692953+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:32:07.692953+0000","last_undegraded":"2023-10-20T10:32:07.692953+0000","last_fullsized":"2023-10-20T10:32:07.692953+0000","mapping_epoch":16,"log_start":"131'502424","ondisk_log_start":"131'502424","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":1530,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 202s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7096635411,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":508402,"num_read_kb":2002396,"num_write":507872,"num_write_kb":2031482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":214,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:30:58.223459+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:30:58.223459+0000","last_peered":"2023-10-20T10:30:58.223459+0000","last_clean":"2023-10-20T10:30:58.223459+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:30:58.223459+0000","last_undegraded":"2023-10-20T10:30:58.223459+0000","last_fullsized":"2023-10-20T10:30:58.223459+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":308,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:30:58.223516+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:30:58.223516+0000","last_peered":"2023-10-20T10:30:58.223516+0000","last_clean":"2023-10-20T10:30:58.223516+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:30:58.223516+0000","last_undegraded":"2023-10-20T10:30:58.223516+0000","last_fullsized":"2023-10-20T10:30:58.223516+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28185780243,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":2016132,"num_read_kb":7941396,"num_write":2015041,"num_write_kb":8060158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16505823232,"data_stored":29888577574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40143,"ondisk_log_size":40143,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":30918,"ondisk_log_size":30918,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739313,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9912100,"kb_used_data":8059996,"kb_used_omap":1747,"kb_used_meta":1850348,"kb_avail":84459740,"statfs":{"total":96636764160,"available":86486773760,"internally_reserved":0,"allocated":8253435904,"data_stored":14943924138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1789111,"internal_metadata":1894757193},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:31:11 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.45600000000000002,"15min":0.48799999999999999},"min":{"1min":0.189,"5min":0.16300000000000001,"15min":0.16200000000000001},"max":{"1min":4.3650000000000002,"5min":6.2430000000000003,"15min":8.2449999999999992},"last":0.377},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.50800000000000001,"15min":0.498},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.18099999999999999},"max":{"1min":4.3840000000000003,"5min":5.048,"15min":6.8239999999999998},"last":0.27700000000000002}]},{"osd":1,"last update":"Fri Oct 20 10:31:11 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.622,"15min":0.60299999999999998},"min":{"1min":0.21099999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.60199999999999998,"5min":6.1509999999999998,"15min":8.282},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.54200000000000004,"15min":0.47799999999999998},"min":{"1min":0.251,"5min":0.23400000000000001,"15min":0.214},"max":{"1min":1.55,"5min":6.109,"15min":6.109},"last":0.35499999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739313,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5699404,"kb_used_data":3953476,"kb_used_omap":678,"kb_used_meta":1745241,"kb_avail":88672436,"statfs":{"total":96636764160,"available":90800574464,"internally_reserved":0,"allocated":4048359424,"data_stored":7312509726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":694339,"internal_metadata":1787127741},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,1,1,9,6,11,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:31:10 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.48399999999999999,"15min":0.47899999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.17199999999999999},"max":{"1min":0.51600000000000001,"5min":4.9480000000000004,"15min":4.9480000000000004},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.48599999999999999,"15min":0.45600000000000002},"min":{"1min":0.21099999999999999,"5min":0.19400000000000001,"15min":0.186},"max":{"1min":1.4099999999999999,"5min":5.0049999999999999,"15min":5.8479999999999999},"last":0.28599999999999998}]},{"osd":2,"last update":"Fri Oct 20 10:31:10 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.51800000000000002,"15min":0.46500000000000002},"min":{"1min":0.22700000000000001,"5min":0.19400000000000001,"15min":0.18099999999999999},"max":{"1min":1.3140000000000001,"5min":7.6459999999999999,"15min":7.6459999999999999},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.435,"5min":0.56100000000000005,"15min":0.50900000000000001},"min":{"1min":0.28199999999999997,"5min":0.253,"15min":0.23799999999999999},"max":{"1min":1.3420000000000001,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.26800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739315,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6119960,"kb_used_data":4109584,"kb_used_omap":700,"kb_used_meta":2009667,"kb_avail":88251880,"statfs":{"total":96636764160,"available":90369925120,"internally_reserved":0,"allocated":4208214016,"data_stored":7633343631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":716869,"internal_metadata":2057899963},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:31:08 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.63900000000000001,"15min":0.58799999999999997},"min":{"1min":0.182,"5min":0.182,"15min":0.17599999999999999},"max":{"1min":0.504,"5min":6.5449999999999999,"15min":9.266},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.502,"15min":0.51000000000000001},"min":{"1min":0.223,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.55600000000000005,"5min":6.2229999999999999,"15min":7.2690000000000001},"last":0.28399999999999997}]},{"osd":2,"last update":"Fri Oct 20 10:31:08 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.62,"15min":0.53400000000000003},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":0.96199999999999997,"5min":6.3029999999999999,"15min":11.176},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.49099999999999999,"15min":0.503},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20000000000000001},"max":{"1min":0.94699999999999995,"5min":4.4260000000000002,"15min":11.093},"last":0.246}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4206784512,"data_stored":7632887808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4046852096,"data_stored":7312125971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8252186624,"data_stored":14943563795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:32:12.605 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:32:12.605 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:32:32.606 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:32:33.030 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:32:33.031 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:32:33.061 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2501,"stamp":"2023-10-20T10:32:31.347291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28218708483,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2063002,"num_read_kb":8006793,"num_write":2023943,"num_write_kb":8094023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71166,"ondisk_log_size":71166,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22023844,"kb_used_data":16186636,"kb_used_omap":3125,"kb_used_meta":5834058,"kb_avail":261091676,"statfs":{"total":289910292480,"available":267357876224,"internally_reserved":0,"allocated":16575115264,"data_stored":29954871760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3200319,"internal_metadata":5974076097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,6,5,12,17,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16199680,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4105,"num_read_kb":16376,"num_write":3960,"num_write_kb":15840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002208"},"pg_stats":[{"pgid":"2.7","version":"133'10248","reported_seq":15208,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:32:27.386420+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:32:27.386420+0000","last_peered":"2023-10-20T10:32:27.386420+0000","last_clean":"2023-10-20T10:32:27.386420+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:32:27.386420+0000","last_undegraded":"2023-10-20T10:32:27.386420+0000","last_fullsized":"2023-10-20T10:32:27.386420+0000","mapping_epoch":13,"log_start":"42'200","ondisk_log_start":"42'200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"133'20156","reported_seq":31052,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:32:27.402783+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:32:27.402783+0000","last_peered":"2023-10-20T10:32:27.402783+0000","last_clean":"2023-10-20T10:32:27.402783+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:32:27.402783+0000","last_undegraded":"2023-10-20T10:32:27.402783+0000","last_fullsized":"2023-10-20T10:32:27.402783+0000","mapping_epoch":13,"log_start":"111'10100","ondisk_log_start":"111'10100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":216,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:12.377083+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:31:12.377083+0000","last_peered":"2023-10-20T10:31:12.377083+0000","last_clean":"2023-10-20T10:31:12.377083+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:31:12.377083+0000","last_undegraded":"2023-10-20T10:31:12.377083+0000","last_fullsized":"2023-10-20T10:31:12.377083+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"133'10209","reported_seq":15944,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:32:27.381191+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:32:27.381191+0000","last_peered":"2023-10-20T10:32:27.381191+0000","last_clean":"2023-10-20T10:32:27.381191+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:32:27.381191+0000","last_undegraded":"2023-10-20T10:32:27.381191+0000","last_fullsized":"2023-10-20T10:32:27.381191+0000","mapping_epoch":13,"log_start":"42'200","ondisk_log_start":"42'200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"133'526414","reported_seq":1043267,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:32:27.711613+0000","last_change":"2023-10-20T10:31:19.489466+0000","last_active":"2023-10-20T10:32:27.711613+0000","last_peered":"2023-10-20T10:32:27.711613+0000","last_clean":"2023-10-20T10:32:27.711613+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:32:27.711613+0000","last_undegraded":"2023-10-20T10:32:27.711613+0000","last_fullsized":"2023-10-20T10:32:27.711613+0000","mapping_epoch":16,"log_start":"131'516332","ondisk_log_start":"131'516332","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'519713","last_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_deep_scrub":"132'519713","last_deep_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_clean_scrub_stamp":"2023-10-20T10:31:19.489275+0000","objects_scrubbed":4836,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":696,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:01:22.001899+0000","scrub_duration":695.57415461999994,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7249166336,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":519714,"num_read_kb":2048648,"num_write":519202,"num_write_kb":2076808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":310,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:12.465812+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:31:12.465812+0000","last_peered":"2023-10-20T10:31:12.465812+0000","last_clean":"2023-10-20T10:31:12.465812+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:31:12.465812+0000","last_undegraded":"2023-10-20T10:31:12.465812+0000","last_fullsized":"2023-10-20T10:31:12.465812+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"133'488770","reported_seq":968250,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:32:27.782922+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:32:27.782922+0000","last_peered":"2023-10-20T10:32:27.782922+0000","last_clean":"2023-10-20T10:32:27.782922+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:32:27.782922+0000","last_undegraded":"2023-10-20T10:32:27.782922+0000","last_fullsized":"2023-10-20T10:32:27.782922+0000","mapping_epoch":16,"log_start":"131'478742","ondisk_log_start":"131'478742","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6723289088,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":482179,"num_read_kb":1898516,"num_write":482026,"num_write_kb":1928104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6801,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:32:23.449053+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:32:23.449053+0000","last_peered":"2023-10-20T10:32:23.449053+0000","last_clean":"2023-10-20T10:32:23.449053+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:32:23.449053+0000","last_undegraded":"2023-10-20T10:32:23.449053+0000","last_fullsized":"2023-10-20T10:32:23.449053+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"133'519059","reported_seq":1028116,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:32:27.716773+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:32:27.716773+0000","last_peered":"2023-10-20T10:32:27.716773+0000","last_clean":"2023-10-20T10:32:27.716773+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:32:27.716773+0000","last_undegraded":"2023-10-20T10:32:27.716773+0000","last_fullsized":"2023-10-20T10:32:27.716773+0000","mapping_epoch":16,"log_start":"131'509036","ondisk_log_start":"131'509036","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7140814848,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":511961,"num_read_kb":2016248,"num_write":511821,"num_write_kb":2047284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6649,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:32:23.480343+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:32:23.480343+0000","last_peered":"2023-10-20T10:32:23.480343+0000","last_clean":"2023-10-20T10:32:23.480343+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:32:23.480343+0000","last_undegraded":"2023-10-20T10:32:23.480343+0000","last_fullsized":"2023-10-20T10:32:23.480343+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"133'514606","reported_seq":1022148,"reported_epoch":133,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:32:27.794188+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:32:27.794188+0000","last_peered":"2023-10-20T10:32:27.794188+0000","last_clean":"2023-10-20T10:32:27.794188+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:32:27.794188+0000","last_undegraded":"2023-10-20T10:32:27.794188+0000","last_fullsized":"2023-10-20T10:32:27.794188+0000","mapping_epoch":16,"log_start":"131'504525","ondisk_log_start":"131'504525","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":1638,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 223s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7104843795,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":510477,"num_read_kb":2010660,"num_write":509880,"num_write_kb":2039514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":215,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:12.377060+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:31:12.377060+0000","last_peered":"2023-10-20T10:31:12.377060+0000","last_clean":"2023-10-20T10:31:12.377060+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:31:12.377060+0000","last_undegraded":"2023-10-20T10:31:12.377060+0000","last_fullsized":"2023-10-20T10:31:12.377060+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":309,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:32:02.640586+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:32:02.640586+0000","last_peered":"2023-10-20T10:32:02.640586+0000","last_clean":"2023-10-20T10:32:02.640586+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:32:02.640586+0000","last_undegraded":"2023-10-20T10:32:02.640586+0000","last_fullsized":"2023-10-20T10:32:02.640586+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28218114067,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":2024331,"num_read_kb":7974072,"num_write":2022929,"num_write_kb":8091710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16570515456,"data_stored":29953269798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40214,"ondisk_log_size":40214,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":30853,"ondisk_log_size":30853,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739317,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10060428,"kb_used_data":8091716,"kb_used_omap":1747,"kb_used_meta":1966956,"kb_avail":84311412,"statfs":{"total":96636764160,"available":86334885888,"internally_reserved":0,"allocated":8285917184,"data_stored":14976401533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1789111,"internal_metadata":2014163785},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:32:15 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.54400000000000004,"15min":0.51600000000000001},"min":{"1min":0.184,"5min":0.184,"15min":0.16200000000000001},"max":{"1min":4.8380000000000001,"5min":6.2430000000000003,"15min":8.2449999999999992},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.26900000000000002,"5min":0.20200000000000001,"15min":0.18099999999999999},"max":{"1min":4.8259999999999996,"5min":4.8259999999999996,"15min":6.8239999999999998},"last":0.371}]},{"osd":1,"last update":"Fri Oct 20 10:32:15 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.47999999999999998,"15min":0.57999999999999996},"min":{"1min":0.23200000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.61599999999999999,"5min":6.0800000000000001,"15min":8.282},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.46999999999999997,"15min":0.47399999999999998},"min":{"1min":0.24199999999999999,"5min":0.23799999999999999,"15min":0.214},"max":{"1min":0.56799999999999995,"5min":6.109,"15min":6.109},"last":0.34000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739317,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5769420,"kb_used_data":3969348,"kb_used_omap":678,"kb_used_meta":1799385,"kb_avail":88602420,"statfs":{"total":96636764160,"available":90728878080,"internally_reserved":0,"allocated":4064612352,"data_stored":7328758769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":694339,"internal_metadata":1842571197},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,8,11,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:32:15 2023","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.41299999999999998,"15min":0.47699999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17199999999999999},"max":{"1min":0.57599999999999996,"5min":3,"15min":4.9480000000000004},"last":0.25},{"interface":"front","average":{"1min":0.255,"5min":0.36699999999999999,"15min":0.438},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.39700000000000002,"5min":4.5010000000000003,"15min":5.8479999999999999},"last":0.50600000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:32:15 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52900000000000003,"15min":0.48099999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.19400000000000001},"max":{"1min":4.2539999999999996,"5min":7.6459999999999999,"15min":7.6459999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57199999999999995,"15min":0.52400000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.23999999999999999},"max":{"1min":4.383,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.56200000000000006}]}]},{"osd":0,"up_from":8,"seq":34359739319,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6193996,"kb_used_data":4125572,"kb_used_omap":700,"kb_used_meta":2067715,"kb_avail":88177844,"statfs":{"total":96636764160,"available":90294112256,"internally_reserved":0,"allocated":4224585728,"data_stored":7649711458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":716869,"internal_metadata":2117341115},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:32:08 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.59199999999999997,"15min":0.59399999999999997},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.17599999999999999},"max":{"1min":2.5059999999999998,"5min":6.5449999999999999,"15min":9.266},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.59199999999999997,"5min":6.2229999999999999,"15min":7.2690000000000001},"last":0.25900000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:32:08 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.63300000000000001,"15min":0.54100000000000004},"min":{"1min":0.255,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":1.391,"5min":6.3029999999999999,"15min":11.176},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.48899999999999999,"15min":0.50700000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20000000000000001},"max":{"1min":1.4330000000000001,"5min":4.4260000000000002,"15min":11.093},"last":0.22600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4223070208,"data_stored":7649173504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4062842880,"data_stored":7328116755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8284602368,"data_stored":14975979539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:32:33.063 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:32:33.064 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:32:36.907 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:32:36.906+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:32:37.344 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:32:37.330+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:32:37.345 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:32:37.330+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:32:53.065 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:32:53.570 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:32:53.571 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:32:53.608 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2512,"stamp":"2023-10-20T10:32:53.357489+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28258169347,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2072732,"num_read_kb":8045497,"num_write":2033576,"num_write_kb":8132555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71265,"ondisk_log_size":71265,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22305776,"kb_used_data":16264216,"kb_used_omap":3125,"kb_used_meta":6038410,"kb_avail":260809744,"statfs":{"total":289910292480,"available":267069177856,"internally_reserved":0,"allocated":16654557184,"data_stored":30034313680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3200319,"internal_metadata":6183332545},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,9,14,13,10,7,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24047616,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5933,"num_read_kb":23604,"num_write":5878,"num_write_kb":23512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008523"},"pg_stats":[{"pgid":"2.7","version":"133'10301","reported_seq":15276,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:32:49.038329+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:32:49.038329+0000","last_peered":"2023-10-20T10:32:49.038329+0000","last_clean":"2023-10-20T10:32:49.038329+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:32:49.038329+0000","last_undegraded":"2023-10-20T10:32:49.038329+0000","last_fullsized":"2023-10-20T10:32:49.038329+0000","mapping_epoch":13,"log_start":"42'300","ondisk_log_start":"42'300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"133'20260","reported_seq":31186,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:32:49.039094+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:32:49.039094+0000","last_peered":"2023-10-20T10:32:49.039094+0000","last_clean":"2023-10-20T10:32:49.039094+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:32:49.039094+0000","last_undegraded":"2023-10-20T10:32:49.039094+0000","last_fullsized":"2023-10-20T10:32:49.039094+0000","mapping_epoch":13,"log_start":"111'10200","ondisk_log_start":"111'10200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":216,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:12.377083+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:31:12.377083+0000","last_peered":"2023-10-20T10:31:12.377083+0000","last_clean":"2023-10-20T10:31:12.377083+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:31:12.377083+0000","last_undegraded":"2023-10-20T10:31:12.377083+0000","last_fullsized":"2023-10-20T10:31:12.377083+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"133'10262","reported_seq":16012,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:32:49.038587+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:32:49.038587+0000","last_peered":"2023-10-20T10:32:49.038587+0000","last_clean":"2023-10-20T10:32:49.038587+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:32:49.038587+0000","last_undegraded":"2023-10-20T10:32:49.038587+0000","last_fullsized":"2023-10-20T10:32:49.038587+0000","mapping_epoch":13,"log_start":"42'200","ondisk_log_start":"42'200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"133'528824","reported_seq":1048152,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:32:52.823558+0000","last_change":"2023-10-20T10:31:19.489466+0000","last_active":"2023-10-20T10:32:52.823558+0000","last_peered":"2023-10-20T10:32:52.823558+0000","last_clean":"2023-10-20T10:32:52.823558+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:32:52.823558+0000","last_undegraded":"2023-10-20T10:32:52.823558+0000","last_fullsized":"2023-10-20T10:32:52.823558+0000","mapping_epoch":16,"log_start":"132'518732","ondisk_log_start":"132'518732","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'519713","last_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_deep_scrub":"132'519713","last_deep_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_clean_scrub_stamp":"2023-10-20T10:31:19.489275+0000","objects_scrubbed":4836,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":696,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:01:22.001899+0000","scrub_duration":695.57415461999994,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7259029504,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":522189,"num_read_kb":2058484,"num_write":521612,"num_write_kb":2086448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":310,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:12.465812+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:31:12.465812+0000","last_peered":"2023-10-20T10:31:12.465812+0000","last_clean":"2023-10-20T10:31:12.465812+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:31:12.465812+0000","last_undegraded":"2023-10-20T10:31:12.465812+0000","last_fullsized":"2023-10-20T10:31:12.465812+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"133'491139","reported_seq":972877,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:32:52.916520+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:32:52.916520+0000","last_peered":"2023-10-20T10:32:52.916520+0000","last_clean":"2023-10-20T10:32:52.916520+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:32:52.916520+0000","last_undegraded":"2023-10-20T10:32:52.916520+0000","last_fullsized":"2023-10-20T10:32:52.916520+0000","mapping_epoch":16,"log_start":"132'481042","ondisk_log_start":"132'481042","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6733000704,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":484437,"num_read_kb":1907480,"num_write":484395,"num_write_kb":1937580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6821,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:32:49.038803+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:32:49.038803+0000","last_peered":"2023-10-20T10:32:49.038803+0000","last_clean":"2023-10-20T10:32:49.038803+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:32:49.038803+0000","last_undegraded":"2023-10-20T10:32:49.038803+0000","last_fullsized":"2023-10-20T10:32:49.038803+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"133'521530","reported_seq":1033116,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:32:52.827061+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:32:52.827061+0000","last_peered":"2023-10-20T10:32:52.827061+0000","last_clean":"2023-10-20T10:32:52.827061+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:32:52.827061+0000","last_undegraded":"2023-10-20T10:32:52.827061+0000","last_fullsized":"2023-10-20T10:32:52.827061+0000","mapping_epoch":16,"log_start":"132'511436","ondisk_log_start":"132'511436","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7150972928,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":514490,"num_read_kb":2026324,"num_write":514292,"num_write_kb":2057168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6669,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:32:49.041238+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:32:49.041238+0000","last_peered":"2023-10-20T10:32:49.041238+0000","last_clean":"2023-10-20T10:32:49.041238+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:32:49.041238+0000","last_undegraded":"2023-10-20T10:32:49.041238+0000","last_fullsized":"2023-10-20T10:32:49.041238+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"133'517145","reported_seq":1027176,"reported_epoch":133,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:32:52.927322+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:32:52.927322+0000","last_peered":"2023-10-20T10:32:52.927322+0000","last_clean":"2023-10-20T10:32:52.927322+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:32:52.927322+0000","last_undegraded":"2023-10-20T10:32:52.927322+0000","last_fullsized":"2023-10-20T10:32:52.927322+0000","mapping_epoch":16,"log_start":"132'507125","ondisk_log_start":"132'507125","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":1848,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 248s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7114571795,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":512945,"num_read_kb":2020488,"num_write":512263,"num_write_kb":2049046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":215,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:12.377060+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:31:12.377060+0000","last_peered":"2023-10-20T10:31:12.377060+0000","last_clean":"2023-10-20T10:31:12.377060+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:31:12.377060+0000","last_undegraded":"2023-10-20T10:31:12.377060+0000","last_fullsized":"2023-10-20T10:31:12.377060+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":309,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:32:02.640586+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:32:02.640586+0000","last_peered":"2023-10-20T10:32:02.640586+0000","last_clean":"2023-10-20T10:32:02.640586+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:32:02.640586+0000","last_undegraded":"2023-10-20T10:32:02.640586+0000","last_fullsized":"2023-10-20T10:32:02.640586+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28257574931,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":2034061,"num_read_kb":8012776,"num_write":2032562,"num_write_kb":8130242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16649224192,"data_stored":30031978534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40303,"ondisk_log_size":40303,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":30863,"ondisk_log_size":30863,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739322,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10222688,"kb_used_data":8130136,"kb_used_omap":1747,"kb_used_meta":2090796,"kb_avail":84149152,"statfs":{"total":96636764160,"available":86168731648,"internally_reserved":0,"allocated":8325259264,"data_stored":15015743613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1789111,"internal_metadata":2140975945},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,4,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:32:15 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.54400000000000004,"15min":0.51600000000000001},"min":{"1min":0.184,"5min":0.184,"15min":0.16200000000000001},"max":{"1min":4.8380000000000001,"5min":6.2430000000000003,"15min":8.2449999999999992},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.26900000000000002,"5min":0.20200000000000001,"15min":0.18099999999999999},"max":{"1min":4.8259999999999996,"5min":4.8259999999999996,"15min":6.8239999999999998},"last":0.31900000000000001}]},{"osd":1,"last update":"Fri Oct 20 10:32:15 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.47999999999999998,"15min":0.57999999999999996},"min":{"1min":0.23200000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.61599999999999999,"5min":6.0800000000000001,"15min":8.282},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.46999999999999997,"15min":0.47399999999999998},"min":{"1min":0.24199999999999999,"5min":0.23799999999999999,"15min":0.214},"max":{"1min":0.56799999999999995,"5min":6.109,"15min":6.109},"last":5.1420000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739322,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5828780,"kb_used_data":3988452,"kb_used_omap":678,"kb_used_meta":1839641,"kb_avail":88543060,"statfs":{"total":96636764160,"available":90668093440,"internally_reserved":0,"allocated":4084174848,"data_stored":7348321265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":694339,"internal_metadata":1883793341},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,3,0,7,7,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:32:15 2023","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.41299999999999998,"15min":0.47699999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17199999999999999},"max":{"1min":0.57599999999999996,"5min":3,"15min":4.9480000000000004},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.255,"5min":0.36699999999999999,"15min":0.438},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.39700000000000002,"5min":4.5010000000000003,"15min":5.8479999999999999},"last":0.23400000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:32:15 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52900000000000003,"15min":0.48099999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.19400000000000001},"max":{"1min":4.2539999999999996,"5min":7.6459999999999999,"15min":7.6459999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57199999999999995,"15min":0.52400000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.23999999999999999},"max":{"1min":4.383,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.59699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739324,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6254308,"kb_used_data":4145628,"kb_used_omap":700,"kb_used_meta":2107971,"kb_avail":88117532,"statfs":{"total":96636764160,"available":90232352768,"internally_reserved":0,"allocated":4245123072,"data_stored":7670248802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":716869,"internal_metadata":2158563259},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,4,7,9,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:32:08 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.59199999999999997,"15min":0.59399999999999997},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.17599999999999999},"max":{"1min":2.5059999999999998,"5min":6.5449999999999999,"15min":9.266},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.216,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.59199999999999997,"5min":6.2229999999999999,"15min":7.2690000000000001},"last":0.25800000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:32:08 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.63300000000000001,"15min":0.54100000000000004},"min":{"1min":0.255,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":1.391,"5min":6.3029999999999999,"15min":11.176},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.48899999999999999,"15min":0.50700000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20000000000000001},"max":{"1min":1.4330000000000001,"5min":4.4260000000000002,"15min":11.093},"last":0.42799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4243062784,"data_stored":7669166080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4082323456,"data_stored":7347597331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8323837952,"data_stored":15015215123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:32:53.610 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:32:53.610 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:33:08.995 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:33:08.990+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:33:09.004 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:33:09.002+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:33:09.036 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:33:09.034+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:33:13.611 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:33:14.074 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:33:14.075 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:33:14.102 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2522,"stamp":"2023-10-20T10:33:13.363765+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28289790467,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2080391,"num_read_kb":8075965,"num_write":2041315,"num_write_kb":8163511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71219,"ondisk_log_size":71219,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21978552,"kb_used_data":16325856,"kb_used_omap":3450,"kb_used_meta":5649221,"kb_avail":261136968,"statfs":{"total":289910292480,"available":267404255232,"internally_reserved":0,"allocated":16717676544,"data_stored":30097433040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3533427,"internal_metadata":5784802701},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,5,5,6,6,7,9,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23994368,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5785,"num_read_kb":23012,"num_write":5879,"num_write_kb":23516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001938"},"pg_stats":[{"pgid":"2.7","version":"133'10353","reported_seq":15340,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:33:12.058233+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:33:12.058233+0000","last_peered":"2023-10-20T10:33:12.058233+0000","last_clean":"2023-10-20T10:33:12.058233+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:33:12.058233+0000","last_undegraded":"2023-10-20T10:33:12.058233+0000","last_fullsized":"2023-10-20T10:33:12.058233+0000","mapping_epoch":13,"log_start":"42'300","ondisk_log_start":"42'300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"133'20363","reported_seq":31313,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:33:12.065373+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:33:12.065373+0000","last_peered":"2023-10-20T10:33:12.065373+0000","last_clean":"2023-10-20T10:33:12.065373+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:33:12.065373+0000","last_undegraded":"2023-10-20T10:33:12.065373+0000","last_fullsized":"2023-10-20T10:33:12.065373+0000","mapping_epoch":13,"log_start":"111'10300","ondisk_log_start":"111'10300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":216,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:12.377083+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:31:12.377083+0000","last_peered":"2023-10-20T10:31:12.377083+0000","last_clean":"2023-10-20T10:31:12.377083+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:31:12.377083+0000","last_undegraded":"2023-10-20T10:31:12.377083+0000","last_fullsized":"2023-10-20T10:31:12.377083+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"133'10314","reported_seq":16076,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:33:12.058029+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:33:12.058029+0000","last_peered":"2023-10-20T10:33:12.058029+0000","last_clean":"2023-10-20T10:33:12.058029+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:33:12.058029+0000","last_undegraded":"2023-10-20T10:33:12.058029+0000","last_fullsized":"2023-10-20T10:33:12.058029+0000","mapping_epoch":13,"log_start":"42'300","ondisk_log_start":"42'300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"133'530816","reported_seq":1052125,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:33:12.869686+0000","last_change":"2023-10-20T10:31:19.489466+0000","last_active":"2023-10-20T10:33:12.869686+0000","last_peered":"2023-10-20T10:33:12.869686+0000","last_clean":"2023-10-20T10:33:12.869686+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:33:12.869686+0000","last_undegraded":"2023-10-20T10:33:12.869686+0000","last_fullsized":"2023-10-20T10:33:12.869686+0000","mapping_epoch":16,"log_start":"132'520732","ondisk_log_start":"132'520732","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'519713","last_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_deep_scrub":"132'519713","last_deep_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_clean_scrub_stamp":"2023-10-20T10:31:19.489275+0000","objects_scrubbed":4836,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":696,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:01:22.001899+0000","scrub_duration":695.57415461999994,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7267192832,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":524170,"num_read_kb":2066364,"num_write":523604,"num_write_kb":2094416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":310,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:12.465812+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:31:12.465812+0000","last_peered":"2023-10-20T10:31:12.465812+0000","last_clean":"2023-10-20T10:31:12.465812+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:31:12.465812+0000","last_undegraded":"2023-10-20T10:31:12.465812+0000","last_fullsized":"2023-10-20T10:31:12.465812+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"133'493004","reported_seq":976613,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:33:13.066790+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:33:13.066790+0000","last_peered":"2023-10-20T10:33:13.066790+0000","last_clean":"2023-10-20T10:33:13.066790+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:33:13.066790+0000","last_undegraded":"2023-10-20T10:33:13.066790+0000","last_fullsized":"2023-10-20T10:33:13.066790+0000","mapping_epoch":16,"log_start":"132'482942","ondisk_log_start":"132'482942","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6740586496,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":486305,"num_read_kb":1914896,"num_write":486260,"num_write_kb":1945040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6838,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:33:09.518418+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:33:09.518418+0000","last_peered":"2023-10-20T10:33:09.518418+0000","last_clean":"2023-10-20T10:33:09.518418+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:33:09.518418+0000","last_undegraded":"2023-10-20T10:33:09.518418+0000","last_fullsized":"2023-10-20T10:33:09.518418+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"133'523477","reported_seq":1036981,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:33:12.853371+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:33:12.853371+0000","last_peered":"2023-10-20T10:33:12.853371+0000","last_clean":"2023-10-20T10:33:12.853371+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:33:12.853371+0000","last_undegraded":"2023-10-20T10:33:12.853371+0000","last_fullsized":"2023-10-20T10:33:12.853371+0000","mapping_epoch":16,"log_start":"132'513436","ondisk_log_start":"132'513436","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7158939648,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":516407,"num_read_kb":2033956,"num_write":516239,"num_write_kb":2064956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6685,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:33:09.518447+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:33:09.518447+0000","last_peered":"2023-10-20T10:33:09.518447+0000","last_clean":"2023-10-20T10:33:09.518447+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:33:09.518447+0000","last_undegraded":"2023-10-20T10:33:09.518447+0000","last_fullsized":"2023-10-20T10:33:09.518447+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"133'519188","reported_seq":1031120,"reported_epoch":133,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:33:13.063830+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:33:13.063830+0000","last_peered":"2023-10-20T10:33:13.063830+0000","last_clean":"2023-10-20T10:33:13.063830+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:33:13.063830+0000","last_undegraded":"2023-10-20T10:33:13.063830+0000","last_fullsized":"2023-10-20T10:33:13.063830+0000","mapping_epoch":16,"log_start":"132'509125","ondisk_log_start":"132'509125","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":1997,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 268s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7122477075,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":514838,"num_read_kb":2028028,"num_write":514198,"num_write_kb":2056786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":215,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:31:12.377060+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:31:12.377060+0000","last_peered":"2023-10-20T10:31:12.377060+0000","last_clean":"2023-10-20T10:31:12.377060+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:31:12.377060+0000","last_undegraded":"2023-10-20T10:31:12.377060+0000","last_fullsized":"2023-10-20T10:31:12.377060+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":309,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-20T10:32:02.640586+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:32:02.640586+0000","last_peered":"2023-10-20T10:32:02.640586+0000","last_clean":"2023-10-20T10:32:02.640586+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:32:02.640586+0000","last_undegraded":"2023-10-20T10:32:02.640586+0000","last_fullsized":"2023-10-20T10:32:02.640586+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28289196051,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":2041720,"num_read_kb":8043244,"num_write":2040301,"num_write_kb":8161198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16712663040,"data_stored":30095417382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40250,"ondisk_log_size":40250,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55322,"internal_metadata":0},"log_size":30870,"ondisk_log_size":30870,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739326,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10374244,"kb_used_data":8160924,"kb_used_omap":1747,"kb_used_meta":2211564,"kb_avail":83997596,"statfs":{"total":96636764160,"available":86013538304,"internally_reserved":0,"allocated":8356786176,"data_stored":15047270525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1789111,"internal_metadata":2264642377},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:32:15 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.54400000000000004,"15min":0.51600000000000001},"min":{"1min":0.184,"5min":0.184,"15min":0.16200000000000001},"max":{"1min":4.8380000000000001,"5min":6.2430000000000003,"15min":8.2449999999999992},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.26900000000000002,"5min":0.20200000000000001,"15min":0.18099999999999999},"max":{"1min":4.8259999999999996,"5min":4.8259999999999996,"15min":6.8239999999999998},"last":0.35199999999999998}]},{"osd":1,"last update":"Fri Oct 20 10:32:15 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.47999999999999998,"15min":0.57999999999999996},"min":{"1min":0.23200000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.61599999999999999,"5min":6.0800000000000001,"15min":8.282},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.46999999999999997,"15min":0.47399999999999998},"min":{"1min":0.24199999999999999,"5min":0.23799999999999999,"15min":0.214},"max":{"1min":0.56799999999999995,"5min":6.109,"15min":6.109},"last":0.38100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739326,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5832336,"kb_used_data":4003592,"kb_used_omap":678,"kb_used_meta":1828057,"kb_avail":88539504,"statfs":{"total":96636764160,"available":90664452096,"internally_reserved":0,"allocated":4099678208,"data_stored":7363824625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":694339,"internal_metadata":1871931325},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,4,7,9,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:32:15 2023","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.41299999999999998,"15min":0.47699999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17199999999999999},"max":{"1min":0.57599999999999996,"5min":3,"15min":4.9480000000000004},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.255,"5min":0.36699999999999999,"15min":0.438},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.39700000000000002,"5min":4.5010000000000003,"15min":5.8479999999999999},"last":0.30199999999999999}]},{"osd":2,"last update":"Fri Oct 20 10:32:15 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52900000000000003,"15min":0.48099999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.19400000000000001},"max":{"1min":4.2539999999999996,"5min":7.6459999999999999,"15min":7.6459999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57199999999999995,"15min":0.52400000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.23999999999999999},"max":{"1min":4.383,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.41099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739328,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5771972,"kb_used_data":4161340,"kb_used_omap":1025,"kb_used_meta":1609598,"kb_avail":88599868,"statfs":{"total":96636764160,"available":90726264832,"internally_reserved":0,"allocated":4261212160,"data_stored":7686337890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1049977,"internal_metadata":1648228999},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.49099999999999999,"15min":0.56699999999999995},"min":{"1min":0.188,"5min":0.18099999999999999,"15min":0.17799999999999999},"max":{"1min":0.49099999999999999,"5min":6.5449999999999999,"15min":8.0869999999999997},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.40999999999999998,"15min":0.47799999999999998},"min":{"1min":0.222,"5min":0.20699999999999999,"15min":0.19800000000000001},"max":{"1min":0.46500000000000002,"5min":6.024,"15min":7.2690000000000001},"last":0.254}]},{"osd":2,"last update":"Fri Oct 20 10:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":0.49199999999999999,"5min":6.3029999999999999,"15min":11.176},"last":0.315},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.433,"15min":0.47999999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.63900000000000001,"5min":4.0220000000000002,"15min":11.093},"last":0.20000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15801,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4259201024,"data_stored":7685304320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4097875968,"data_stored":7363149843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8355586048,"data_stored":15046963219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:33:14.104 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:33:14.104 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:33:30.211 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:33:30.201+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:33:30.211 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:33:30.201+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:33:34.105 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:33:34.728 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:33:34.729 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:33:34.761 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2533,"stamp":"2023-10-20T10:33:33.367216+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28322181635,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2088389,"num_read_kb":8107753,"num_write":2049243,"num_write_kb":8195223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71153,"ondisk_log_size":71153,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20858680,"kb_used_data":16387552,"kb_used_omap":3673,"kb_used_meta":4467430,"kb_avail":262256840,"statfs":{"total":289910292480,"available":268551004160,"internally_reserved":0,"allocated":16780853248,"data_stored":30160597960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3761851,"internal_metadata":4574648645},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,6,5,7,10,16,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24211456,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5908,"num_read_kb":23488,"num_write":5926,"num_write_kb":23704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002180"},"pg_stats":[{"pgid":"2.7","version":"134'10381","reported_seq":15381,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:30.215215+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:33:30.215215+0000","last_peered":"2023-10-20T10:33:30.215215+0000","last_clean":"2023-10-20T10:33:30.215215+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:33:30.215215+0000","last_undegraded":"2023-10-20T10:33:30.215215+0000","last_fullsized":"2023-10-20T10:33:30.215215+0000","mapping_epoch":13,"log_start":"42'300","ondisk_log_start":"42'300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"134'20418","reported_seq":31393,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:30.266458+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:33:30.266458+0000","last_peered":"2023-10-20T10:33:30.266458+0000","last_clean":"2023-10-20T10:33:30.266458+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:33:30.266458+0000","last_undegraded":"2023-10-20T10:33:30.266458+0000","last_fullsized":"2023-10-20T10:33:30.266458+0000","mapping_epoch":13,"log_start":"113'10400","ondisk_log_start":"113'10400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":218,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:33:18.111773+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:33:18.111773+0000","last_peered":"2023-10-20T10:33:18.111773+0000","last_clean":"2023-10-20T10:33:18.111773+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:33:18.111773+0000","last_undegraded":"2023-10-20T10:33:18.111773+0000","last_fullsized":"2023-10-20T10:33:18.111773+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"134'10342","reported_seq":16118,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:30.213828+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:33:30.213828+0000","last_peered":"2023-10-20T10:33:30.213828+0000","last_clean":"2023-10-20T10:33:30.213828+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:33:30.213828+0000","last_undegraded":"2023-10-20T10:33:30.213828+0000","last_fullsized":"2023-10-20T10:33:30.213828+0000","mapping_epoch":13,"log_start":"42'300","ondisk_log_start":"42'300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"134'532892","reported_seq":1056218,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:32.921807+0000","last_change":"2023-10-20T10:31:19.489466+0000","last_active":"2023-10-20T10:33:32.921807+0000","last_peered":"2023-10-20T10:33:32.921807+0000","last_clean":"2023-10-20T10:33:32.921807+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:33:32.921807+0000","last_undegraded":"2023-10-20T10:33:32.921807+0000","last_fullsized":"2023-10-20T10:33:32.921807+0000","mapping_epoch":16,"log_start":"132'522832","ondisk_log_start":"132'522832","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'519713","last_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_deep_scrub":"132'519713","last_deep_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_clean_scrub_stamp":"2023-10-20T10:31:19.489275+0000","objects_scrubbed":4836,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":696,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:01:22.001899+0000","scrub_duration":695.57415461999994,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7275663360,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":526186,"num_read_kb":2074384,"num_write":525680,"num_write_kb":2102720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":312,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:33:17.899774+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:33:17.899774+0000","last_peered":"2023-10-20T10:33:17.899774+0000","last_clean":"2023-10-20T10:33:17.899774+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:33:17.899774+0000","last_undegraded":"2023-10-20T10:33:17.899774+0000","last_fullsized":"2023-10-20T10:33:17.899774+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"134'494846","reported_seq":980330,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:33.160905+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:33:33.160905+0000","last_peered":"2023-10-20T10:33:33.160905+0000","last_clean":"2023-10-20T10:33:33.160905+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:33:33.160905+0000","last_undegraded":"2023-10-20T10:33:33.160905+0000","last_fullsized":"2023-10-20T10:33:33.160905+0000","mapping_epoch":16,"log_start":"132'484842","ondisk_log_start":"132'484842","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6748127232,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":488179,"num_read_kb":1922344,"num_write":488102,"num_write_kb":1952408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6855,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:30.004452+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:33:30.004452+0000","last_peered":"2023-10-20T10:33:30.004452+0000","last_clean":"2023-10-20T10:33:30.004452+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:33:30.004452+0000","last_undegraded":"2023-10-20T10:33:30.004452+0000","last_fullsized":"2023-10-20T10:33:30.004452+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"134'525472","reported_seq":1041064,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:32.922102+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:33:32.922102+0000","last_peered":"2023-10-20T10:33:32.922102+0000","last_clean":"2023-10-20T10:33:32.922102+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:33:32.922102+0000","last_undegraded":"2023-10-20T10:33:32.922102+0000","last_fullsized":"2023-10-20T10:33:32.922102+0000","mapping_epoch":16,"log_start":"132'515436","ondisk_log_start":"132'515436","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7167082496,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":518494,"num_read_kb":2042264,"num_write":518234,"num_write_kb":2072936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6702,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:30.001824+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:33:30.001824+0000","last_peered":"2023-10-20T10:33:30.001824+0000","last_clean":"2023-10-20T10:33:30.001824+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:33:30.001824+0000","last_undegraded":"2023-10-20T10:33:30.001824+0000","last_fullsized":"2023-10-20T10:33:30.001824+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"134'521298","reported_seq":1035267,"reported_epoch":134,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:33:33.181560+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:33:33.181560+0000","last_peered":"2023-10-20T10:33:33.181560+0000","last_clean":"2023-10-20T10:33:33.181560+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:33:33.181560+0000","last_undegraded":"2023-10-20T10:33:33.181560+0000","last_fullsized":"2023-10-20T10:33:33.181560+0000","mapping_epoch":16,"log_start":"132'511225","ondisk_log_start":"132'511225","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":2120,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 288s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7130714131,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":516859,"num_read_kb":2036040,"num_write":516213,"num_write_kb":2064846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":217,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:33:18.111752+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:33:18.111752+0000","last_peered":"2023-10-20T10:33:18.111752+0000","last_clean":"2023-10-20T10:33:18.111752+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:33:18.111752+0000","last_undegraded":"2023-10-20T10:33:18.111752+0000","last_fullsized":"2023-10-20T10:33:18.111752+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":310,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:32:08.266892+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:32:08.266892+0000","last_peered":"2023-10-20T10:32:08.266892+0000","last_clean":"2023-10-20T10:32:08.266892+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:32:08.266892+0000","last_undegraded":"2023-10-20T10:32:08.266892+0000","last_fullsized":"2023-10-20T10:32:08.266892+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28321587219,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":2049718,"num_read_kb":8075032,"num_write":2048229,"num_write_kb":8192910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16777568256,"data_stored":30160322598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40173,"ondisk_log_size":40173,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53442,"internal_metadata":0},"log_size":30881,"ondisk_log_size":30881,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739330,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9132856,"kb_used_data":8191280,"kb_used_omap":1970,"kb_used_meta":939597,"kb_avail":85238984,"statfs":{"total":96636764160,"available":87284719616,"internally_reserved":0,"allocated":8387870720,"data_stored":15078351141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2017535,"internal_metadata":962148097},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.55100000000000005,"15min":0.495},"min":{"1min":0.19400000000000001,"5min":0.184,"15min":0.16200000000000001},"max":{"1min":2.419,"5min":6.2430000000000003,"15min":6.8419999999999996},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.50700000000000001,"15min":0.52500000000000002},"min":{"1min":0.254,"5min":0.20200000000000001,"15min":0.18099999999999999},"max":{"1min":2.4350000000000001,"5min":4.8259999999999996,"15min":6.8239999999999998},"last":0.27100000000000002}]},{"osd":1,"last update":"Fri Oct 20 10:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.47799999999999998,"15min":0.54400000000000004},"min":{"1min":0.216,"5min":0.20699999999999999,"15min":0.186},"max":{"1min":2.2320000000000002,"5min":6.0800000000000001,"15min":8.282},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.878,"5min":0.55100000000000005,"15min":0.497},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.214},"max":{"1min":5.1420000000000003,"5min":6.109,"15min":6.109},"last":0.308}]}]},{"osd":1,"up_from":8,"seq":34359739330,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5865744,"kb_used_data":4018952,"kb_used_omap":678,"kb_used_meta":1846105,"kb_avail":88506096,"statfs":{"total":96636764160,"available":90630242304,"internally_reserved":0,"allocated":4115406848,"data_stored":7379549337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":694339,"internal_metadata":1890412477},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,1,1,1,7,16,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:33:20 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.39600000000000002,"15min":0.46100000000000002},"min":{"1min":0.18099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":1.849,"5min":2.7839999999999998,"15min":4.9480000000000004},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.36499999999999999,"15min":0.42599999999999999},"min":{"1min":0.21199999999999999,"5min":0.185,"15min":0.185},"max":{"1min":1.6539999999999999,"5min":4.5010000000000003,"15min":5.4640000000000004},"last":0.23000000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:33:20 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.505,"15min":0.46999999999999997},"min":{"1min":0.218,"5min":0.20699999999999999,"15min":0.19400000000000001},"max":{"1min":3.1749999999999998,"5min":4.2539999999999996,"15min":7.6459999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55000000000000004,"15min":0.51400000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.24099999999999999},"max":{"1min":3.206,"5min":4.383,"15min":7.6820000000000004},"last":0.47199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739332,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5860080,"kb_used_data":4177320,"kb_used_omap":1025,"kb_used_meta":1681726,"kb_avail":88511760,"statfs":{"total":96636764160,"available":90636042240,"internally_reserved":0,"allocated":4277575680,"data_stored":7702697482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1049977,"internal_metadata":1722088071},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.49099999999999999,"15min":0.56699999999999995},"min":{"1min":0.188,"5min":0.18099999999999999,"15min":0.17799999999999999},"max":{"1min":0.49099999999999999,"5min":6.5449999999999999,"15min":8.0869999999999997},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.40999999999999998,"15min":0.47799999999999998},"min":{"1min":0.222,"5min":0.20699999999999999,"15min":0.19800000000000001},"max":{"1min":0.46500000000000002,"5min":6.024,"15min":7.2690000000000001},"last":0.30599999999999999}]},{"osd":2,"last update":"Fri Oct 20 10:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":0.49199999999999999,"5min":6.3029999999999999,"15min":11.176},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.433,"15min":0.47999999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.63900000000000001,"5min":4.0220000000000002,"15min":11.093},"last":0.33100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13921,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4275871744,"data_stored":7701975040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4113678336,"data_stored":7378952211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8388018176,"data_stored":15079395347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:33:34.763 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:33:34.764 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-20T10:33:35.345 INFO:teuthology.orchestra.run.smithi150.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-20T10:33:35.377 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:33:38.579 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:33:38.577+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:33:38.590 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:33:38.585+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T10:33:38.590 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:33:38.585+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:33:38.605 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:33:38.597+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab4f70 2023-10-20T10:33:38.605 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:33:38.597+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T10:33:38.605 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:33:38.597+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:33:38.606 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:33:38.597+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T10:33:38.606 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:33:38.597+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T10:33:38.606 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:33:38.597+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:33:38.607 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:33:38.597+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:33:38.607 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:33:38.601+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2750 2023-10-20T10:33:55.379 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:33:55.837 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:33:55.838 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:33:55.866 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2544,"stamp":"2023-10-20T10:33:55.371153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28353958403,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2096215,"num_read_kb":8138893,"num_write":2057030,"num_write_kb":8226371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71299,"ondisk_log_size":71299,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20385452,"kb_used_data":16451092,"kb_used_omap":3898,"kb_used_meta":3930437,"kb_avail":262730068,"statfs":{"total":289910292480,"available":269035589632,"internally_reserved":0,"allocated":16845918208,"data_stored":30225662920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3991741,"internal_metadata":4024768323},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,3,10,7,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16412672,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4035,"num_read_kb":16060,"num_write":4020,"num_write_kb":16080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002186"},"pg_stats":[{"pgid":"2.7","version":"134'10420","reported_seq":15432,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:51.494155+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:33:51.494155+0000","last_peered":"2023-10-20T10:33:51.494155+0000","last_clean":"2023-10-20T10:33:51.494155+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:33:51.494155+0000","last_undegraded":"2023-10-20T10:33:51.494155+0000","last_fullsized":"2023-10-20T10:33:51.494155+0000","mapping_epoch":13,"log_start":"44'400","ondisk_log_start":"44'400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"134'20496","reported_seq":31495,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:51.513503+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:33:51.513503+0000","last_peered":"2023-10-20T10:33:51.513503+0000","last_clean":"2023-10-20T10:33:51.513503+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:33:51.513503+0000","last_undegraded":"2023-10-20T10:33:51.513503+0000","last_fullsized":"2023-10-20T10:33:51.513503+0000","mapping_epoch":13,"log_start":"113'10400","ondisk_log_start":"113'10400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":218,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:33:18.111773+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:33:18.111773+0000","last_peered":"2023-10-20T10:33:18.111773+0000","last_clean":"2023-10-20T10:33:18.111773+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:33:18.111773+0000","last_undegraded":"2023-10-20T10:33:18.111773+0000","last_fullsized":"2023-10-20T10:33:18.111773+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"134'10381","reported_seq":16169,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:51.505485+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:33:51.505485+0000","last_peered":"2023-10-20T10:33:51.505485+0000","last_clean":"2023-10-20T10:33:51.505485+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:33:51.505485+0000","last_undegraded":"2023-10-20T10:33:51.505485+0000","last_fullsized":"2023-10-20T10:33:51.505485+0000","mapping_epoch":13,"log_start":"42'300","ondisk_log_start":"42'300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"134'534900","reported_seq":1060234,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:53.058171+0000","last_change":"2023-10-20T10:31:19.489466+0000","last_active":"2023-10-20T10:33:53.058171+0000","last_peered":"2023-10-20T10:33:53.058171+0000","last_clean":"2023-10-20T10:33:53.058171+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:33:53.058171+0000","last_undegraded":"2023-10-20T10:33:53.058171+0000","last_fullsized":"2023-10-20T10:33:53.058171+0000","mapping_epoch":16,"log_start":"133'524832","ondisk_log_start":"133'524832","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'519713","last_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_deep_scrub":"132'519713","last_deep_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_clean_scrub_stamp":"2023-10-20T10:31:19.489275+0000","objects_scrubbed":4836,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":696,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:01:22.001899+0000","scrub_duration":695.57415461999994,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7283871744,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":528196,"num_read_kb":2082368,"num_write":527688,"num_write_kb":2110752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":312,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:33:17.899774+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:33:17.899774+0000","last_peered":"2023-10-20T10:33:17.899774+0000","last_clean":"2023-10-20T10:33:17.899774+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:33:17.899774+0000","last_undegraded":"2023-10-20T10:33:17.899774+0000","last_fullsized":"2023-10-20T10:33:17.899774+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"134'496704","reported_seq":984060,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:53.294003+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:33:53.294003+0000","last_peered":"2023-10-20T10:33:53.294003+0000","last_clean":"2023-10-20T10:33:53.294003+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:33:53.294003+0000","last_undegraded":"2023-10-20T10:33:53.294003+0000","last_fullsized":"2023-10-20T10:33:53.294003+0000","mapping_epoch":16,"log_start":"132'486642","ondisk_log_start":"132'486642","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6755713024,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":490052,"num_read_kb":1929796,"num_write":489960,"num_write_kb":1959840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6871,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:50.482575+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:33:50.482575+0000","last_peered":"2023-10-20T10:33:50.482575+0000","last_clean":"2023-10-20T10:33:50.482575+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:33:50.482575+0000","last_undegraded":"2023-10-20T10:33:50.482575+0000","last_fullsized":"2023-10-20T10:33:50.482575+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"134'527414","reported_seq":1044996,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:53.017909+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:33:53.017909+0000","last_peered":"2023-10-20T10:33:53.017909+0000","last_clean":"2023-10-20T10:33:53.017909+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:33:53.017909+0000","last_undegraded":"2023-10-20T10:33:53.017909+0000","last_fullsized":"2023-10-20T10:33:53.017909+0000","mapping_epoch":16,"log_start":"133'517336","ondisk_log_start":"133'517336","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7175004160,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":520485,"num_read_kb":2050208,"num_write":520176,"num_write_kb":2080704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6718,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:50.482905+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:33:50.482905+0000","last_peered":"2023-10-20T10:33:50.482905+0000","last_clean":"2023-10-20T10:33:50.482905+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:33:50.482905+0000","last_undegraded":"2023-10-20T10:33:50.482905+0000","last_fullsized":"2023-10-20T10:33:50.482905+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"134'523380","reported_seq":1039309,"reported_epoch":134,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:33:53.278045+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:33:53.278045+0000","last_peered":"2023-10-20T10:33:53.278045+0000","last_clean":"2023-10-20T10:33:53.278045+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:33:53.278045+0000","last_undegraded":"2023-10-20T10:33:53.278045+0000","last_fullsized":"2023-10-20T10:33:53.278045+0000","mapping_epoch":16,"log_start":"133'513325","ondisk_log_start":"133'513325","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":2257,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 308s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7138775059,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":518811,"num_read_kb":2043800,"num_write":518192,"num_write_kb":2072762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":217,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:33:18.111752+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:33:18.111752+0000","last_peered":"2023-10-20T10:33:18.111752+0000","last_clean":"2023-10-20T10:33:18.111752+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:33:18.111752+0000","last_undegraded":"2023-10-20T10:33:18.111752+0000","last_fullsized":"2023-10-20T10:33:18.111752+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":310,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:32:08.266892+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:32:08.266892+0000","last_peered":"2023-10-20T10:32:08.266892+0000","last_clean":"2023-10-20T10:32:08.266892+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:32:08.266892+0000","last_undegraded":"2023-10-20T10:32:08.266892+0000","last_fullsized":"2023-10-20T10:32:08.266892+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28353363987,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":2057544,"num_read_kb":8106172,"num_write":2056016,"num_write_kb":8224058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16841236480,"data_stored":30223990822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40263,"ondisk_log_size":40263,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53442,"internal_metadata":0},"log_size":30937,"ondisk_log_size":30937,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739334,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9217292,"kb_used_data":8223748,"kb_used_omap":1970,"kb_used_meta":991565,"kb_avail":85154548,"statfs":{"total":96636764160,"available":87198257152,"internally_reserved":0,"allocated":8421117952,"data_stored":15111598373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2017535,"internal_metadata":1015363329},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.55100000000000005,"15min":0.495},"min":{"1min":0.19400000000000001,"5min":0.184,"15min":0.16200000000000001},"max":{"1min":2.419,"5min":6.2430000000000003,"15min":6.8419999999999996},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.50700000000000001,"15min":0.52500000000000002},"min":{"1min":0.254,"5min":0.20200000000000001,"15min":0.18099999999999999},"max":{"1min":2.4350000000000001,"5min":4.8259999999999996,"15min":6.8239999999999998},"last":0.27100000000000002}]},{"osd":1,"last update":"Fri Oct 20 10:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.47799999999999998,"15min":0.54400000000000004},"min":{"1min":0.216,"5min":0.20699999999999999,"15min":0.186},"max":{"1min":2.2320000000000002,"5min":6.0800000000000001,"15min":8.282},"last":0.308},{"interface":"front","average":{"1min":0.878,"5min":0.55100000000000005,"15min":0.497},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.214},"max":{"1min":5.1420000000000003,"5min":6.109,"15min":6.109},"last":0.23400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739334,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5249608,"kb_used_data":4034240,"kb_used_omap":902,"kb_used_meta":1214457,"kb_avail":89122232,"statfs":{"total":96636764160,"available":91261165568,"internally_reserved":0,"allocated":4131061760,"data_stored":7395204249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":924229,"internal_metadata":1243604411},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,6,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:33:20 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.39600000000000002,"15min":0.46100000000000002},"min":{"1min":0.18099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":1.849,"5min":2.7839999999999998,"15min":4.9480000000000004},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.36499999999999999,"15min":0.42599999999999999},"min":{"1min":0.21199999999999999,"5min":0.185,"15min":0.185},"max":{"1min":1.6539999999999999,"5min":4.5010000000000003,"15min":5.4640000000000004},"last":0.251}]},{"osd":2,"last update":"Fri Oct 20 10:33:20 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.505,"15min":0.46999999999999997},"min":{"1min":0.218,"5min":0.20699999999999999,"15min":0.19400000000000001},"max":{"1min":3.1749999999999998,"5min":4.2539999999999996,"15min":7.6459999999999999},"last":0.372},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55000000000000004,"15min":0.51400000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.24099999999999999},"max":{"1min":3.206,"5min":4.383,"15min":7.6820000000000004},"last":0.51200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739336,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5918552,"kb_used_data":4193104,"kb_used_omap":1025,"kb_used_meta":1724414,"kb_avail":88453288,"statfs":{"total":96636764160,"available":90576166912,"internally_reserved":0,"allocated":4293738496,"data_stored":7718860298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1049977,"internal_metadata":1765800583},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.49099999999999999,"15min":0.56699999999999995},"min":{"1min":0.188,"5min":0.18099999999999999,"15min":0.17799999999999999},"max":{"1min":0.49099999999999999,"5min":6.5449999999999999,"15min":8.0869999999999997},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.40999999999999998,"15min":0.47799999999999998},"min":{"1min":0.222,"5min":0.20699999999999999,"15min":0.19800000000000001},"max":{"1min":0.46500000000000002,"5min":6.024,"15min":7.2690000000000001},"last":0.317}]},{"osd":2,"last update":"Fri Oct 20 10:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":0.49199999999999999,"5min":6.3029999999999999,"15min":11.176},"last":0.223},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.433,"15min":0.47999999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.63900000000000001,"5min":4.0220000000000002,"15min":11.093},"last":0.44}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13921,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4292059136,"data_stored":7718162432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4129394688,"data_stored":7394668563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8419782656,"data_stored":15111159827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:33:55.869 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:33:55.869 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:34:15.871 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:34:16.419 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:34:16.420 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:34:16.475 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2554,"stamp":"2023-10-20T10:34:15.376497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28385280515,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2103884,"num_read_kb":8169393,"num_write":2064701,"num_write_kb":8257055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71110,"ondisk_log_size":71110,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20628380,"kb_used_data":16512644,"kb_used_omap":3898,"kb_used_meta":4111813,"kb_avail":262487140,"statfs":{"total":289910292480,"available":268786831360,"internally_reserved":0,"allocated":16908947456,"data_stored":30288692168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3991741,"internal_metadata":4210497347},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,3,6,4,5,6,12,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15843328,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3912,"num_read_kb":15564,"num_write":3881,"num_write_kb":15524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001987"},"pg_stats":[{"pgid":"2.7","version":"134'10459","reported_seq":15483,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:34:12.955652+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:34:12.955652+0000","last_peered":"2023-10-20T10:34:12.955652+0000","last_clean":"2023-10-20T10:34:12.955652+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:34:12.955652+0000","last_undegraded":"2023-10-20T10:34:12.955652+0000","last_fullsized":"2023-10-20T10:34:12.955652+0000","mapping_epoch":13,"log_start":"44'400","ondisk_log_start":"44'400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"134'20574","reported_seq":31598,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:34:12.969592+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:34:12.969592+0000","last_peered":"2023-10-20T10:34:12.969592+0000","last_clean":"2023-10-20T10:34:12.969592+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:34:12.969592+0000","last_undegraded":"2023-10-20T10:34:12.969592+0000","last_fullsized":"2023-10-20T10:34:12.969592+0000","mapping_epoch":13,"log_start":"113'10500","ondisk_log_start":"113'10500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":218,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:33:18.111773+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:33:18.111773+0000","last_peered":"2023-10-20T10:33:18.111773+0000","last_clean":"2023-10-20T10:33:18.111773+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:33:18.111773+0000","last_undegraded":"2023-10-20T10:33:18.111773+0000","last_fullsized":"2023-10-20T10:33:18.111773+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"134'10420","reported_seq":16220,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:34:12.957624+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:34:12.957624+0000","last_peered":"2023-10-20T10:34:12.957624+0000","last_clean":"2023-10-20T10:34:12.957624+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:34:12.957624+0000","last_undegraded":"2023-10-20T10:34:12.957624+0000","last_fullsized":"2023-10-20T10:34:12.957624+0000","mapping_epoch":13,"log_start":"44'400","ondisk_log_start":"44'400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"134'536862","reported_seq":1064151,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:34:13.083177+0000","last_change":"2023-10-20T10:31:19.489466+0000","last_active":"2023-10-20T10:34:13.083177+0000","last_peered":"2023-10-20T10:34:13.083177+0000","last_clean":"2023-10-20T10:34:13.083177+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:34:13.083177+0000","last_undegraded":"2023-10-20T10:34:13.083177+0000","last_fullsized":"2023-10-20T10:34:13.083177+0000","mapping_epoch":16,"log_start":"133'526832","ondisk_log_start":"133'526832","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'519713","last_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_deep_scrub":"132'519713","last_deep_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_clean_scrub_stamp":"2023-10-20T10:31:19.489275+0000","objects_scrubbed":4836,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":696,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:01:22.001899+0000","scrub_duration":695.57415461999994,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7291908096,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":530148,"num_read_kb":2090136,"num_write":529650,"num_write_kb":2118600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":312,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:33:17.899774+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:33:17.899774+0000","last_peered":"2023-10-20T10:33:17.899774+0000","last_clean":"2023-10-20T10:33:17.899774+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:33:17.899774+0000","last_undegraded":"2023-10-20T10:33:17.899774+0000","last_fullsized":"2023-10-20T10:33:17.899774+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"134'498474","reported_seq":987657,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:34:13.369260+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:34:13.369260+0000","last_peered":"2023-10-20T10:34:13.369260+0000","last_clean":"2023-10-20T10:34:13.369260+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:34:13.369260+0000","last_undegraded":"2023-10-20T10:34:13.369260+0000","last_fullsized":"2023-10-20T10:34:13.369260+0000","mapping_epoch":16,"log_start":"133'488442","ondisk_log_start":"133'488442","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6762917888,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":491881,"num_read_kb":1937076,"num_write":491730,"num_write_kb":1966920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6887,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:34:10.958408+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:34:10.958408+0000","last_peered":"2023-10-20T10:34:10.958408+0000","last_clean":"2023-10-20T10:34:10.958408+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:34:10.958408+0000","last_undegraded":"2023-10-20T10:34:10.958408+0000","last_fullsized":"2023-10-20T10:34:10.958408+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"134'529343","reported_seq":1048878,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:34:13.077901+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:34:13.077901+0000","last_peered":"2023-10-20T10:34:13.077901+0000","last_clean":"2023-10-20T10:34:13.077901+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:34:13.077901+0000","last_undegraded":"2023-10-20T10:34:13.077901+0000","last_fullsized":"2023-10-20T10:34:13.077901+0000","mapping_epoch":16,"log_start":"133'519336","ondisk_log_start":"133'519336","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7182876672,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":522438,"num_read_kb":2057980,"num_write":522105,"num_write_kb":2088420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6735,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:34:11.005132+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:34:11.005132+0000","last_peered":"2023-10-20T10:34:11.005132+0000","last_clean":"2023-10-20T10:34:11.005132+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:34:11.005132+0000","last_undegraded":"2023-10-20T10:34:11.005132+0000","last_fullsized":"2023-10-20T10:34:11.005132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"134'525474","reported_seq":1043355,"reported_epoch":134,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:34:13.369350+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:34:13.369350+0000","last_peered":"2023-10-20T10:34:13.369350+0000","last_clean":"2023-10-20T10:34:13.369350+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:34:13.369350+0000","last_undegraded":"2023-10-20T10:34:13.369350+0000","last_fullsized":"2023-10-20T10:34:13.369350+0000","mapping_epoch":16,"log_start":"133'515425","ondisk_log_start":"133'515425","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":2375,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 328s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7146983443,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":520746,"num_read_kb":2051480,"num_write":520202,"num_write_kb":2080802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":217,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-20T10:33:18.111752+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:33:18.111752+0000","last_peered":"2023-10-20T10:33:18.111752+0000","last_clean":"2023-10-20T10:33:18.111752+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:33:18.111752+0000","last_undegraded":"2023-10-20T10:33:18.111752+0000","last_fullsized":"2023-10-20T10:33:18.111752+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":311,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:18.283185+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:33:18.283185+0000","last_peered":"2023-10-20T10:33:18.283185+0000","last_clean":"2023-10-20T10:33:18.283185+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:33:18.283185+0000","last_undegraded":"2023-10-20T10:33:18.283185+0000","last_fullsized":"2023-10-20T10:33:18.283185+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28384686099,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":2065213,"num_read_kb":8136672,"num_write":2063687,"num_write_kb":8254742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16904372224,"data_stored":30287126566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40118,"ondisk_log_size":40118,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53442,"internal_metadata":0},"log_size":30893,"ondisk_log_size":30893,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739338,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9332740,"kb_used_data":8254524,"kb_used_omap":1970,"kb_used_meta":1076237,"kb_avail":85039100,"statfs":{"total":96636764160,"available":87080038400,"internally_reserved":0,"allocated":8452632576,"data_stored":15143112997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2017535,"internal_metadata":1102067457},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.55100000000000005,"15min":0.495},"min":{"1min":0.19400000000000001,"5min":0.184,"15min":0.16200000000000001},"max":{"1min":2.419,"5min":6.2430000000000003,"15min":6.8419999999999996},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.50700000000000001,"15min":0.52500000000000002},"min":{"1min":0.254,"5min":0.20200000000000001,"15min":0.18099999999999999},"max":{"1min":2.4350000000000001,"5min":4.8259999999999996,"15min":6.8239999999999998},"last":0.28999999999999998}]},{"osd":1,"last update":"Fri Oct 20 10:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.47799999999999998,"15min":0.54400000000000004},"min":{"1min":0.216,"5min":0.20699999999999999,"15min":0.186},"max":{"1min":2.2320000000000002,"5min":6.0800000000000001,"15min":8.282},"last":0.25},{"interface":"front","average":{"1min":0.878,"5min":0.55100000000000005,"15min":0.497},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.214},"max":{"1min":5.1420000000000003,"5min":6.109,"15min":6.109},"last":0.46100000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739338,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5307416,"kb_used_data":4049424,"kb_used_omap":902,"kb_used_meta":1257081,"kb_avail":89064424,"statfs":{"total":96636764160,"available":91201970176,"internally_reserved":0,"allocated":4146610176,"data_stored":7410752665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":924229,"internal_metadata":1287251387},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,6,2,4,6,12,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:33:20 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.39600000000000002,"15min":0.46100000000000002},"min":{"1min":0.18099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":1.849,"5min":2.7839999999999998,"15min":4.9480000000000004},"last":5.0750000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.36499999999999999,"15min":0.42599999999999999},"min":{"1min":0.21199999999999999,"5min":0.185,"15min":0.185},"max":{"1min":1.6539999999999999,"5min":4.5010000000000003,"15min":5.4640000000000004},"last":0.20899999999999999}]},{"osd":2,"last update":"Fri Oct 20 10:33:20 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.505,"15min":0.46999999999999997},"min":{"1min":0.218,"5min":0.20699999999999999,"15min":0.19400000000000001},"max":{"1min":3.1749999999999998,"5min":4.2539999999999996,"15min":7.6459999999999999},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55000000000000004,"15min":0.51400000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.24099999999999999},"max":{"1min":3.206,"5min":4.383,"15min":7.6820000000000004},"last":0.254}]}]},{"osd":0,"up_from":8,"seq":34359739340,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5988224,"kb_used_data":4208696,"kb_used_omap":1025,"kb_used_meta":1778494,"kb_avail":88383616,"statfs":{"total":96636764160,"available":90504822784,"internally_reserved":0,"allocated":4309704704,"data_stored":7734826506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1049977,"internal_metadata":1821178503},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.49099999999999999,"15min":0.56699999999999995},"min":{"1min":0.188,"5min":0.18099999999999999,"15min":0.17799999999999999},"max":{"1min":0.49099999999999999,"5min":6.5449999999999999,"15min":8.0869999999999997},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.40999999999999998,"15min":0.47799999999999998},"min":{"1min":0.222,"5min":0.20699999999999999,"15min":0.19800000000000001},"max":{"1min":0.46500000000000002,"5min":6.024,"15min":7.2690000000000001},"last":0.26500000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":0.49199999999999999,"5min":6.3029999999999999,"15min":11.176},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.433,"15min":0.47999999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.63900000000000001,"5min":4.0220000000000002,"15min":11.093},"last":0.26000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13921,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4307988480,"data_stored":7734091776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4144869376,"data_stored":7410143251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8451514368,"data_stored":15142891539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:34:16.477 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:34:16.478 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:34:36.478 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:34:36.997 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:34:36.999 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:34:37.030 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2565,"stamp":"2023-10-20T10:34:35.384981+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28416782851,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2111567,"num_read_kb":8199965,"num_write":2072420,"num_write_kb":8287931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71152,"ondisk_log_size":71152,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20854884,"kb_used_data":16574348,"kb_used_omap":3898,"kb_used_meta":4276613,"kb_avail":262260636,"statfs":{"total":289910292480,"available":268554891264,"internally_reserved":0,"allocated":16972132352,"data_stored":30351865151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3991741,"internal_metadata":4379252547},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,3,5,9,8,9,11,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19673088,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4871,"num_read_kb":19376,"num_write":4822,"num_write_kb":19288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007200"},"pg_stats":[{"pgid":"2.7","version":"135'10504","reported_seq":15542,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:34:31.442306+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:34:31.442306+0000","last_peered":"2023-10-20T10:34:31.442306+0000","last_clean":"2023-10-20T10:34:31.442306+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:34:31.442306+0000","last_undegraded":"2023-10-20T10:34:31.442306+0000","last_fullsized":"2023-10-20T10:34:31.442306+0000","mapping_epoch":13,"log_start":"44'500","ondisk_log_start":"44'500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"135'20664","reported_seq":31713,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:34:31.443270+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:34:31.443270+0000","last_peered":"2023-10-20T10:34:31.443270+0000","last_clean":"2023-10-20T10:34:31.443270+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:34:31.443270+0000","last_undegraded":"2023-10-20T10:34:31.443270+0000","last_fullsized":"2023-10-20T10:34:31.443270+0000","mapping_epoch":13,"log_start":"115'10600","ondisk_log_start":"115'10600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":219,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:18.282954+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:33:18.282954+0000","last_peered":"2023-10-20T10:33:18.282954+0000","last_clean":"2023-10-20T10:33:18.282954+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:33:18.282954+0000","last_undegraded":"2023-10-20T10:33:18.282954+0000","last_fullsized":"2023-10-20T10:33:18.282954+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"135'10465","reported_seq":16278,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:34:31.442702+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:34:31.442702+0000","last_peered":"2023-10-20T10:34:31.442702+0000","last_clean":"2023-10-20T10:34:31.442702+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:34:31.442702+0000","last_undegraded":"2023-10-20T10:34:31.442702+0000","last_fullsized":"2023-10-20T10:34:31.442702+0000","mapping_epoch":13,"log_start":"44'400","ondisk_log_start":"44'400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"135'538818","reported_seq":1068068,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:34:33.124328+0000","last_change":"2023-10-20T10:31:19.489466+0000","last_active":"2023-10-20T10:34:33.124328+0000","last_peered":"2023-10-20T10:34:33.124328+0000","last_clean":"2023-10-20T10:34:33.124328+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:34:33.124328+0000","last_undegraded":"2023-10-20T10:34:33.124328+0000","last_fullsized":"2023-10-20T10:34:33.124328+0000","mapping_epoch":16,"log_start":"133'528732","ondisk_log_start":"133'528732","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'519713","last_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_deep_scrub":"132'519713","last_deep_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_clean_scrub_stamp":"2023-10-20T10:31:19.489275+0000","objects_scrubbed":4836,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":696,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:01:22.001899+0000","scrub_duration":695.57415461999994,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7299883008,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":532110,"num_read_kb":2097940,"num_write":531606,"num_write_kb":2126424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":313,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:18.267505+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:33:18.267505+0000","last_peered":"2023-10-20T10:33:18.267505+0000","last_clean":"2023-10-20T10:33:18.267505+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:33:18.267505+0000","last_undegraded":"2023-10-20T10:33:18.267505+0000","last_fullsized":"2023-10-20T10:33:18.267505+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"135'500269","reported_seq":991309,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:34:33.475504+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:34:33.475504+0000","last_peered":"2023-10-20T10:34:33.475504+0000","last_clean":"2023-10-20T10:34:33.475504+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:34:33.475504+0000","last_undegraded":"2023-10-20T10:34:33.475504+0000","last_fullsized":"2023-10-20T10:34:33.475504+0000","mapping_epoch":16,"log_start":"133'490242","ondisk_log_start":"133'490242","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6770262016,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":493737,"num_read_kb":1944464,"num_write":493525,"num_write_kb":1974100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"127'334","reported_seq":6904,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:34:31.448899+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:34:31.448899+0000","last_peered":"2023-10-20T10:34:31.448899+0000","last_clean":"2023-10-20T10:34:31.448899+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:34:31.448899+0000","last_undegraded":"2023-10-20T10:34:31.448899+0000","last_fullsized":"2023-10-20T10:34:31.448899+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"135'531363","reported_seq":1052875,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:34:33.127786+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:34:33.127786+0000","last_peered":"2023-10-20T10:34:33.127786+0000","last_clean":"2023-10-20T10:34:33.127786+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:34:33.127786+0000","last_undegraded":"2023-10-20T10:34:33.127786+0000","last_fullsized":"2023-10-20T10:34:33.127786+0000","mapping_epoch":16,"log_start":"133'521336","ondisk_log_start":"133'521336","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7191105536,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":524413,"num_read_kb":2065840,"num_write":524125,"num_write_kb":2096500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"127'311","reported_seq":6752,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:34:31.449375+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:34:31.449375+0000","last_peered":"2023-10-20T10:34:31.449375+0000","last_clean":"2023-10-20T10:34:31.449375+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:34:31.449375+0000","last_undegraded":"2023-10-20T10:34:31.449375+0000","last_fullsized":"2023-10-20T10:34:31.449375+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"135'527565","reported_seq":1047351,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:34:33.514763+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:34:33.514763+0000","last_peered":"2023-10-20T10:34:33.514763+0000","last_clean":"2023-10-20T10:34:33.514763+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:34:33.514763+0000","last_undegraded":"2023-10-20T10:34:33.514763+0000","last_fullsized":"2023-10-20T10:34:33.514763+0000","mapping_epoch":16,"log_start":"133'517525","ondisk_log_start":"133'517525","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":2564,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 348s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7154937875,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":522636,"num_read_kb":2059000,"num_write":522150,"num_write_kb":2088594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":218,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:18.282880+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:33:18.282880+0000","last_peered":"2023-10-20T10:33:18.282880+0000","last_clean":"2023-10-20T10:33:18.282880+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:33:18.282880+0000","last_undegraded":"2023-10-20T10:33:18.282880+0000","last_fullsized":"2023-10-20T10:33:18.282880+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":312,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:34:03.341144+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:34:03.341144+0000","last_peered":"2023-10-20T10:34:03.341144+0000","last_clean":"2023-10-20T10:34:03.341144+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:34:03.341144+0000","last_undegraded":"2023-10-20T10:34:03.341144+0000","last_fullsized":"2023-10-20T10:34:03.341144+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28416188435,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":2072896,"num_read_kb":8167244,"num_write":2071406,"num_write_kb":8285618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16967340032,"data_stored":30350094374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40180,"ondisk_log_size":40180,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53442,"internal_metadata":0},"log_size":30873,"ondisk_log_size":30873,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739342,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9449392,"kb_used_data":8285096,"kb_used_omap":1970,"kb_used_meta":1162317,"kb_avail":84922448,"statfs":{"total":96636764160,"available":86960586752,"internally_reserved":0,"allocated":8483938304,"data_stored":15174414754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2017535,"internal_metadata":1190213377},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:34:18 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.48099999999999998,"15min":0.437},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16200000000000001},"max":{"1min":0.52100000000000002,"5min":4.8380000000000001,"15min":6.2430000000000003},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.502,"15min":0.47699999999999998},"min":{"1min":0.14799999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.90900000000000003,"5min":4.8259999999999996,"15min":5.048},"last":0.371}]},{"osd":1,"last update":"Fri Oct 20 10:34:18 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.40000000000000002,"15min":0.495},"min":{"1min":0.191,"5min":0.191,"15min":0.186},"max":{"1min":0.84299999999999997,"5min":3.0790000000000002,"15min":6.1509999999999998},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.37,"5min":0.47399999999999998,"15min":0.48199999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57599999999999996,"5min":5.1420000000000003,"15min":6.109},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739342,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5365492,"kb_used_data":4064876,"kb_used_omap":902,"kb_used_meta":1299705,"kb_avail":89006348,"statfs":{"total":96636764160,"available":91142500352,"internally_reserved":0,"allocated":4162433024,"data_stored":7426571542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":924229,"internal_metadata":1330898363},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,4,3,9,11,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:34:23 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.40400000000000003,"15min":0.46100000000000002},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.0750000000000002,"5min":5.0750000000000002,"15min":5.0750000000000002},"last":0.18099999999999999},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.32400000000000001,"15min":0.42699999999999999},"min":{"1min":0.19700000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.71399999999999997,"5min":1.6539999999999999,"15min":5.4640000000000004},"last":0.21199999999999999}]},{"osd":2,"last update":"Fri Oct 20 10:34:23 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.47999999999999998,"15min":0.48299999999999998},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":4.5899999999999999,"5min":4.5899999999999999,"15min":7.6459999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.254,"5min":0.247,"15min":0.24099999999999999},"max":{"1min":4.625,"5min":4.625,"15min":7.6820000000000004},"last":0.29799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739344,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6040000,"kb_used_data":4224376,"kb_used_omap":1025,"kb_used_meta":1814590,"kb_avail":88331840,"statfs":{"total":96636764160,"available":90451804160,"internally_reserved":0,"allocated":4325761024,"data_stored":7750878855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1049977,"internal_metadata":1858140807},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.49299999999999999,"15min":0.56200000000000006},"min":{"1min":0.19600000000000001,"5min":0.18099999999999999,"15min":0.17799999999999999},"max":{"1min":0.94199999999999995,"5min":6.5449999999999999,"15min":8.0869999999999997},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.314,"5min":0.40600000000000003,"15min":0.48099999999999998},"min":{"1min":0.21199999999999999,"5min":0.20699999999999999,"15min":0.19800000000000001},"max":{"1min":0.54200000000000004,"5min":6.024,"15min":7.2690000000000001},"last":0.28699999999999998}]},{"osd":2,"last update":"Fri Oct 20 10:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.50800000000000001,"15min":0.47099999999999997},"min":{"1min":0.223,"5min":0.223,"15min":0.20200000000000001},"max":{"1min":0.50900000000000001,"5min":6.3029999999999999,"15min":6.3029999999999999},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.45700000000000002,"15min":0.437},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.9660000000000002,"5min":4.0220000000000002,"15min":4.4260000000000002},"last":0.22600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13921,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4324286464,"data_stored":7750389760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4160208896,"data_stored":7425482771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8482844672,"data_stored":15174221843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:34:37.032 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:34:37.033 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:34:57.034 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:34:57.642 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:34:57.643 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:34:57.666 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2576,"stamp":"2023-10-20T10:34:57.393606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28448821763,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2119317,"num_read_kb":8230801,"num_write":2080243,"num_write_kb":8319223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71194,"ondisk_log_size":71194,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21138832,"kb_used_data":16636020,"kb_used_omap":3898,"kb_used_meta":4498885,"kb_avail":261976688,"statfs":{"total":289910292480,"available":268264128512,"internally_reserved":0,"allocated":17035284480,"data_stored":30415017279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3991741,"internal_metadata":4606859075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,1,4,9,11,7,9,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16158720,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3843,"num_read_kb":15276,"num_write":3949,"num_write_kb":15796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006863"},"pg_stats":[{"pgid":"2.7","version":"135'10549","reported_seq":15599,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:34:51.918157+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:34:51.918157+0000","last_peered":"2023-10-20T10:34:51.918157+0000","last_clean":"2023-10-20T10:34:51.918157+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:34:51.918157+0000","last_undegraded":"2023-10-20T10:34:51.918157+0000","last_fullsized":"2023-10-20T10:34:51.918157+0000","mapping_epoch":13,"log_start":"44'500","ondisk_log_start":"44'500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"135'20754","reported_seq":31827,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:34:51.919715+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:34:51.919715+0000","last_peered":"2023-10-20T10:34:51.919715+0000","last_clean":"2023-10-20T10:34:51.919715+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:34:51.919715+0000","last_undegraded":"2023-10-20T10:34:51.919715+0000","last_fullsized":"2023-10-20T10:34:51.919715+0000","mapping_epoch":13,"log_start":"115'10700","ondisk_log_start":"115'10700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":219,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:18.282954+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:33:18.282954+0000","last_peered":"2023-10-20T10:33:18.282954+0000","last_clean":"2023-10-20T10:33:18.282954+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:33:18.282954+0000","last_undegraded":"2023-10-20T10:33:18.282954+0000","last_fullsized":"2023-10-20T10:33:18.282954+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"135'10510","reported_seq":16335,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:34:51.918230+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:34:51.918230+0000","last_peered":"2023-10-20T10:34:51.918230+0000","last_clean":"2023-10-20T10:34:51.918230+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:34:51.918230+0000","last_undegraded":"2023-10-20T10:34:51.918230+0000","last_fullsized":"2023-10-20T10:34:51.918230+0000","mapping_epoch":13,"log_start":"44'500","ondisk_log_start":"44'500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"135'540824","reported_seq":1072004,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:34:53.228803+0000","last_change":"2023-10-20T10:31:19.489466+0000","last_active":"2023-10-20T10:34:53.228803+0000","last_peered":"2023-10-20T10:34:53.228803+0000","last_clean":"2023-10-20T10:34:53.228803+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:34:53.228803+0000","last_undegraded":"2023-10-20T10:34:53.228803+0000","last_fullsized":"2023-10-20T10:34:53.228803+0000","mapping_epoch":16,"log_start":"133'530732","ondisk_log_start":"133'530732","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'519713","last_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_deep_scrub":"132'519713","last_deep_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_clean_scrub_stamp":"2023-10-20T10:31:19.489275+0000","objects_scrubbed":4836,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":696,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:01:22.001899+0000","scrub_duration":695.57415461999994,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7308079104,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":534039,"num_read_kb":2105600,"num_write":533612,"num_write_kb":2134448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":313,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:18.267505+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:33:18.267505+0000","last_peered":"2023-10-20T10:33:18.267505+0000","last_clean":"2023-10-20T10:33:18.267505+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:33:18.267505+0000","last_undegraded":"2023-10-20T10:33:18.267505+0000","last_fullsized":"2023-10-20T10:33:18.267505+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"135'502159","reported_seq":995112,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:34:53.555600+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:34:53.555600+0000","last_peered":"2023-10-20T10:34:53.555600+0000","last_clean":"2023-10-20T10:34:53.555600+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:34:53.555600+0000","last_undegraded":"2023-10-20T10:34:53.555600+0000","last_fullsized":"2023-10-20T10:34:53.555600+0000","mapping_epoch":16,"log_start":"133'492142","ondisk_log_start":"133'492142","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6777999360,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":495649,"num_read_kb":1952080,"num_write":495415,"num_write_kb":1981660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"135'338","reported_seq":6920,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:34:51.918661+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:34:51.918661+0000","last_peered":"2023-10-20T10:34:51.918661+0000","last_clean":"2023-10-20T10:34:51.918661+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:34:51.918661+0000","last_undegraded":"2023-10-20T10:34:51.918661+0000","last_fullsized":"2023-10-20T10:34:51.918661+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"135'533332","reported_seq":1056778,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:34:53.223089+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:34:53.223089+0000","last_peered":"2023-10-20T10:34:53.223089+0000","last_clean":"2023-10-20T10:34:53.223089+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:34:53.223089+0000","last_undegraded":"2023-10-20T10:34:53.223089+0000","last_fullsized":"2023-10-20T10:34:53.223089+0000","mapping_epoch":16,"log_start":"133'523236","ondisk_log_start":"133'523236","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7199186944,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":526346,"num_read_kb":2073528,"num_write":526094,"num_write_kb":2104376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"135'315","reported_seq":6768,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:34:51.937806+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:34:51.937806+0000","last_peered":"2023-10-20T10:34:51.937806+0000","last_clean":"2023-10-20T10:34:51.937806+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:34:51.937806+0000","last_undegraded":"2023-10-20T10:34:51.937806+0000","last_fullsized":"2023-10-20T10:34:51.937806+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"135'529654","reported_seq":1051426,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:34:53.581779+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:34:53.581779+0000","last_peered":"2023-10-20T10:34:53.581779+0000","last_clean":"2023-10-20T10:34:53.581779+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:34:53.581779+0000","last_undegraded":"2023-10-20T10:34:53.581779+0000","last_fullsized":"2023-10-20T10:34:53.581779+0000","mapping_epoch":16,"log_start":"133'519625","ondisk_log_start":"133'519625","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":2732,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 368s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7162961939,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":524612,"num_read_kb":2066872,"num_write":524108,"num_write_kb":2096426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":218,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:18.282880+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:33:18.282880+0000","last_peered":"2023-10-20T10:33:18.282880+0000","last_clean":"2023-10-20T10:33:18.282880+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:33:18.282880+0000","last_undegraded":"2023-10-20T10:33:18.282880+0000","last_fullsized":"2023-10-20T10:33:18.282880+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":312,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:34:03.341144+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:34:03.341144+0000","last_peered":"2023-10-20T10:34:03.341144+0000","last_clean":"2023-10-20T10:34:03.341144+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:34:03.341144+0000","last_undegraded":"2023-10-20T10:34:03.341144+0000","last_fullsized":"2023-10-20T10:34:03.341144+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28448227347,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":2080646,"num_read_kb":8198080,"num_write":2079229,"num_write_kb":8316910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17031397376,"data_stored":30414151718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40234,"ondisk_log_size":40234,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53442,"internal_metadata":0},"log_size":30861,"ondisk_log_size":30861,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739346,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9576532,"kb_used_data":8316236,"kb_used_omap":1970,"kb_used_meta":1258317,"kb_avail":84795308,"statfs":{"total":96636764160,"available":86830395392,"internally_reserved":0,"allocated":8515825664,"data_stored":15206302114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2017535,"internal_metadata":1288517377},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,5,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:34:18 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.48099999999999998,"15min":0.437},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16200000000000001},"max":{"1min":0.52100000000000002,"5min":4.8380000000000001,"15min":6.2430000000000003},"last":0.433},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.502,"15min":0.47699999999999998},"min":{"1min":0.14799999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.90900000000000003,"5min":4.8259999999999996,"15min":5.048},"last":0.49299999999999999}]},{"osd":1,"last update":"Fri Oct 20 10:34:18 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.40000000000000002,"15min":0.495},"min":{"1min":0.191,"5min":0.191,"15min":0.186},"max":{"1min":0.84299999999999997,"5min":3.0790000000000002,"15min":6.1509999999999998},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.37,"5min":0.47399999999999998,"15min":0.48199999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57599999999999996,"5min":5.1420000000000003,"15min":6.109},"last":0.46899999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739346,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5434324,"kb_used_data":4079560,"kb_used_omap":902,"kb_used_meta":1353849,"kb_avail":88937516,"statfs":{"total":96636764160,"available":91072016384,"internally_reserved":0,"allocated":4177469440,"data_stored":7441607958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":924229,"internal_metadata":1386341819},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,2,7,4,9,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:34:23 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.40400000000000003,"15min":0.46100000000000002},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.0750000000000002,"5min":5.0750000000000002,"15min":5.0750000000000002},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.32400000000000001,"15min":0.42699999999999999},"min":{"1min":0.19700000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.71399999999999997,"5min":1.6539999999999999,"15min":5.4640000000000004},"last":0.33000000000000002}]},{"osd":2,"last update":"Fri Oct 20 10:34:23 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.47999999999999998,"15min":0.48299999999999998},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":4.5899999999999999,"5min":4.5899999999999999,"15min":7.6459999999999999},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.254,"5min":0.247,"15min":0.24099999999999999},"max":{"1min":4.625,"5min":4.625,"15min":7.6820000000000004},"last":0.313}]}]},{"osd":0,"up_from":8,"seq":34359739348,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6127976,"kb_used_data":4240224,"kb_used_omap":1025,"kb_used_meta":1886718,"kb_avail":88243864,"statfs":{"total":96636764160,"available":90361716736,"internally_reserved":0,"allocated":4341989376,"data_stored":7767107207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1049977,"internal_metadata":1931999879},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.49299999999999999,"15min":0.56200000000000006},"min":{"1min":0.19600000000000001,"5min":0.18099999999999999,"15min":0.17799999999999999},"max":{"1min":0.94199999999999995,"5min":6.5449999999999999,"15min":8.0869999999999997},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.314,"5min":0.40600000000000003,"15min":0.48099999999999998},"min":{"1min":0.21199999999999999,"5min":0.20699999999999999,"15min":0.19800000000000001},"max":{"1min":0.54200000000000004,"5min":6.024,"15min":7.2690000000000001},"last":0.22800000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.50800000000000001,"15min":0.47099999999999997},"min":{"1min":0.223,"5min":0.223,"15min":0.20200000000000001},"max":{"1min":0.50900000000000001,"5min":6.3029999999999999,"15min":6.3029999999999999},"last":0.371},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.45700000000000002,"15min":0.437},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.9660000000000002,"5min":4.0220000000000002,"15min":4.4260000000000002},"last":0.34599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13921,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4340555776,"data_stored":7766659072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4175982592,"data_stored":7441256467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8514859008,"data_stored":15206236179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:34:57.668 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:34:57.668 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:35:17.670 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:35:18.277 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:35:18.278 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:35:18.312 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2586,"stamp":"2023-10-20T10:35:17.396830+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28480377347,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2127075,"num_read_kb":8261705,"num_write":2087974,"num_write_kb":8350147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71240,"ondisk_log_size":71240,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21407260,"kb_used_data":16697604,"kb_used_omap":3898,"kb_used_meta":4705733,"kb_avail":261708260,"statfs":{"total":289910292480,"available":267989258240,"internally_reserved":0,"allocated":17098346496,"data_stored":30478079295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3991741,"internal_metadata":4818671427},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,4,4,5,4,14,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16003072,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3872,"num_read_kb":15408,"num_write":3913,"num_write_kb":15652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001871"},"pg_stats":[{"pgid":"2.7","version":"135'10600","reported_seq":15662,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:35:12.397972+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:35:12.397972+0000","last_peered":"2023-10-20T10:35:12.397972+0000","last_clean":"2023-10-20T10:35:12.397972+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:35:12.397972+0000","last_undegraded":"2023-10-20T10:35:12.397972+0000","last_fullsized":"2023-10-20T10:35:12.397972+0000","mapping_epoch":13,"log_start":"44'500","ondisk_log_start":"44'500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"135'20856","reported_seq":31953,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:35:12.398844+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:35:12.398844+0000","last_peered":"2023-10-20T10:35:12.398844+0000","last_clean":"2023-10-20T10:35:12.398844+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:35:12.398844+0000","last_undegraded":"2023-10-20T10:35:12.398844+0000","last_fullsized":"2023-10-20T10:35:12.398844+0000","mapping_epoch":13,"log_start":"115'10800","ondisk_log_start":"115'10800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":219,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:18.282954+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:33:18.282954+0000","last_peered":"2023-10-20T10:33:18.282954+0000","last_clean":"2023-10-20T10:33:18.282954+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:33:18.282954+0000","last_undegraded":"2023-10-20T10:33:18.282954+0000","last_fullsized":"2023-10-20T10:33:18.282954+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"135'10561","reported_seq":16398,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:35:12.398067+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:35:12.398067+0000","last_peered":"2023-10-20T10:35:12.398067+0000","last_clean":"2023-10-20T10:35:12.398067+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:35:12.398067+0000","last_undegraded":"2023-10-20T10:35:12.398067+0000","last_fullsized":"2023-10-20T10:35:12.398067+0000","mapping_epoch":13,"log_start":"44'500","ondisk_log_start":"44'500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"135'542850","reported_seq":1076005,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:35:13.300302+0000","last_change":"2023-10-20T10:31:19.489466+0000","last_active":"2023-10-20T10:35:13.300302+0000","last_peered":"2023-10-20T10:35:13.300302+0000","last_clean":"2023-10-20T10:35:13.300302+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:35:13.300302+0000","last_undegraded":"2023-10-20T10:35:13.300302+0000","last_fullsized":"2023-10-20T10:35:13.300302+0000","mapping_epoch":16,"log_start":"134'532832","ondisk_log_start":"134'532832","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'519713","last_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_deep_scrub":"132'519713","last_deep_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_clean_scrub_stamp":"2023-10-20T10:31:19.489275+0000","objects_scrubbed":4836,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":696,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:01:22.001899+0000","scrub_duration":695.57415461999994,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7316340736,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":536013,"num_read_kb":2113468,"num_write":535638,"num_write_kb":2142552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":313,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:18.267505+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:33:18.267505+0000","last_peered":"2023-10-20T10:33:18.267505+0000","last_clean":"2023-10-20T10:33:18.267505+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:33:18.267505+0000","last_undegraded":"2023-10-20T10:33:18.267505+0000","last_fullsized":"2023-10-20T10:33:18.267505+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"135'504028","reported_seq":998873,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:35:13.701653+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:35:13.701653+0000","last_peered":"2023-10-20T10:35:13.701653+0000","last_clean":"2023-10-20T10:35:13.701653+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:35:13.701653+0000","last_undegraded":"2023-10-20T10:35:13.701653+0000","last_fullsized":"2023-10-20T10:35:13.701653+0000","mapping_epoch":16,"log_start":"134'493942","ondisk_log_start":"134'493942","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6785617920,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":497540,"num_read_kb":1959608,"num_write":497284,"num_write_kb":1989136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"135'338","reported_seq":6937,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:35:12.398564+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:35:12.398564+0000","last_peered":"2023-10-20T10:35:12.398564+0000","last_clean":"2023-10-20T10:35:12.398564+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:35:12.398564+0000","last_undegraded":"2023-10-20T10:35:12.398564+0000","last_fullsized":"2023-10-20T10:35:12.398564+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"135'535167","reported_seq":1060604,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:35:13.310428+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:35:13.310428+0000","last_peered":"2023-10-20T10:35:13.310428+0000","last_clean":"2023-10-20T10:35:13.310428+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:35:13.310428+0000","last_undegraded":"2023-10-20T10:35:13.310428+0000","last_fullsized":"2023-10-20T10:35:13.310428+0000","mapping_epoch":16,"log_start":"134'525136","ondisk_log_start":"134'525136","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7206686720,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":528337,"num_read_kb":2081452,"num_write":527929,"num_write_kb":2111716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"135'315","reported_seq":6784,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:35:12.399633+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:35:12.399633+0000","last_peered":"2023-10-20T10:35:12.399633+0000","last_clean":"2023-10-20T10:35:12.399633+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:35:12.399633+0000","last_undegraded":"2023-10-20T10:35:12.399633+0000","last_fullsized":"2023-10-20T10:35:12.399633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"135'531766","reported_seq":1055456,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:35:13.683381+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:35:13.683381+0000","last_peered":"2023-10-20T10:35:13.683381+0000","last_clean":"2023-10-20T10:35:13.683381+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:35:13.683381+0000","last_undegraded":"2023-10-20T10:35:13.683381+0000","last_fullsized":"2023-10-20T10:35:13.683381+0000","mapping_epoch":16,"log_start":"134'521725","ondisk_log_start":"134'521725","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":2876,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 388s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7171137555,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":526514,"num_read_kb":2074456,"num_write":526109,"num_write_kb":2104430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":218,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:33:18.282880+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:33:18.282880+0000","last_peered":"2023-10-20T10:33:18.282880+0000","last_clean":"2023-10-20T10:33:18.282880+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:33:18.282880+0000","last_undegraded":"2023-10-20T10:33:18.282880+0000","last_fullsized":"2023-10-20T10:33:18.282880+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":312,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-20T10:34:03.341144+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:34:03.341144+0000","last_peered":"2023-10-20T10:34:03.341144+0000","last_clean":"2023-10-20T10:34:03.341144+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:34:03.341144+0000","last_undegraded":"2023-10-20T10:34:03.341144+0000","last_fullsized":"2023-10-20T10:34:03.341144+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28479782931,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":2088404,"num_read_kb":8228984,"num_write":2086960,"num_write_kb":8347834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17094578176,"data_stored":30477332518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40176,"ondisk_log_size":40176,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53442,"internal_metadata":0},"log_size":30965,"ondisk_log_size":30965,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739350,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9723768,"kb_used_data":8346864,"kb_used_omap":1970,"kb_used_meta":1374925,"kb_avail":84648072,"statfs":{"total":96636764160,"available":86679625728,"internally_reserved":0,"allocated":8547188736,"data_stored":15237665186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2017535,"internal_metadata":1407923969},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:34:18 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.48099999999999998,"15min":0.437},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16200000000000001},"max":{"1min":0.52100000000000002,"5min":4.8380000000000001,"15min":6.2430000000000003},"last":0.433},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.502,"15min":0.47699999999999998},"min":{"1min":0.14799999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.90900000000000003,"5min":4.8259999999999996,"15min":5.048},"last":0.46600000000000003}]},{"osd":1,"last update":"Fri Oct 20 10:34:18 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.40000000000000002,"15min":0.495},"min":{"1min":0.191,"5min":0.191,"15min":0.186},"max":{"1min":0.84299999999999997,"5min":3.0790000000000002,"15min":6.1509999999999998},"last":0.307},{"interface":"front","average":{"1min":0.37,"5min":0.47399999999999998,"15min":0.48199999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57599999999999996,"5min":5.1420000000000003,"15min":6.109},"last":0.60399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739350,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5503992,"kb_used_data":4095088,"kb_used_omap":902,"kb_used_meta":1407993,"kb_avail":88867848,"statfs":{"total":96636764160,"available":91000676352,"internally_reserved":0,"allocated":4193370112,"data_stored":7457508630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":924229,"internal_metadata":1441785275},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,0,1,4,14,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:34:23 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.40400000000000003,"15min":0.46100000000000002},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.0750000000000002,"5min":5.0750000000000002,"15min":5.0750000000000002},"last":0.18099999999999999},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.32400000000000001,"15min":0.42699999999999999},"min":{"1min":0.19700000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.71399999999999997,"5min":1.6539999999999999,"15min":5.4640000000000004},"last":0.19500000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:34:23 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.47999999999999998,"15min":0.48299999999999998},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":4.5899999999999999,"5min":4.5899999999999999,"15min":7.6459999999999999},"last":0.224},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.254,"5min":0.247,"15min":0.24099999999999999},"max":{"1min":4.625,"5min":4.625,"15min":7.6820000000000004},"last":0.24299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739352,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6179500,"kb_used_data":4255652,"kb_used_omap":1025,"kb_used_meta":1922814,"kb_avail":88192340,"statfs":{"total":96636764160,"available":90308956160,"internally_reserved":0,"allocated":4357787648,"data_stored":7782905479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1049977,"internal_metadata":1968962183},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.49299999999999999,"15min":0.56200000000000006},"min":{"1min":0.19600000000000001,"5min":0.18099999999999999,"15min":0.17799999999999999},"max":{"1min":0.94199999999999995,"5min":6.5449999999999999,"15min":8.0869999999999997},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.314,"5min":0.40600000000000003,"15min":0.48099999999999998},"min":{"1min":0.21199999999999999,"5min":0.20699999999999999,"15min":0.19800000000000001},"max":{"1min":0.54200000000000004,"5min":6.024,"15min":7.2690000000000001},"last":0.28799999999999998}]},{"osd":2,"last update":"Fri Oct 20 10:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.50800000000000001,"15min":0.47099999999999997},"min":{"1min":0.223,"5min":0.223,"15min":0.20200000000000001},"max":{"1min":0.50900000000000001,"5min":6.3029999999999999,"15min":6.3029999999999999},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.45700000000000002,"15min":0.437},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.9660000000000002,"5min":4.0220000000000002,"15min":4.4260000000000002},"last":1.4239999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13921,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4356370432,"data_stored":7782473728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4191813632,"data_stored":7457087507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8546394112,"data_stored":15237771283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:35:18.314 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:35:18.315 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:35:38.316 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:35:38.898 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:35:38.899 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:35:38.947 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2597,"stamp":"2023-10-20T10:35:37.402679+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28511756803,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2134901,"num_read_kb":8292849,"num_write":2095636,"num_write_kb":8380795,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71199,"ondisk_log_size":71199,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21654652,"kb_used_data":16758628,"kb_used_omap":3898,"kb_used_meta":4892101,"kb_avail":261460868,"statfs":{"total":289910292480,"available":267735928832,"internally_reserved":0,"allocated":17160835072,"data_stored":30540567871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3991741,"internal_metadata":5009512259},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,3,7,2,8,7,14,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15626240,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3812,"num_read_kb":15160,"num_write":3809,"num_write_kb":15236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004258"},"pg_stats":[{"pgid":"2.7","version":"135'10645","reported_seq":15720,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:35:32.878158+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:35:32.878158+0000","last_peered":"2023-10-20T10:35:32.878158+0000","last_clean":"2023-10-20T10:35:32.878158+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:35:32.878158+0000","last_undegraded":"2023-10-20T10:35:32.878158+0000","last_fullsized":"2023-10-20T10:35:32.878158+0000","mapping_epoch":13,"log_start":"46'600","ondisk_log_start":"46'600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"135'20946","reported_seq":32068,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:35:32.878832+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:35:32.878832+0000","last_peered":"2023-10-20T10:35:32.878832+0000","last_clean":"2023-10-20T10:35:32.878832+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:35:32.878832+0000","last_undegraded":"2023-10-20T10:35:32.878832+0000","last_fullsized":"2023-10-20T10:35:32.878832+0000","mapping_epoch":13,"log_start":"118'10900","ondisk_log_start":"118'10900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":221,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:35:23.744496+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:35:23.744496+0000","last_peered":"2023-10-20T10:35:23.744496+0000","last_clean":"2023-10-20T10:35:23.744496+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:35:23.744496+0000","last_undegraded":"2023-10-20T10:35:23.744496+0000","last_fullsized":"2023-10-20T10:35:23.744496+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"135'10606","reported_seq":16457,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:35:32.878202+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:35:32.878202+0000","last_peered":"2023-10-20T10:35:32.878202+0000","last_clean":"2023-10-20T10:35:32.878202+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:35:32.878202+0000","last_undegraded":"2023-10-20T10:35:32.878202+0000","last_fullsized":"2023-10-20T10:35:32.878202+0000","mapping_epoch":13,"log_start":"46'600","ondisk_log_start":"46'600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"136'544717","reported_seq":1079751,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:35:33.345534+0000","last_change":"2023-10-20T10:31:19.489466+0000","last_active":"2023-10-20T10:35:33.345534+0000","last_peered":"2023-10-20T10:35:33.345534+0000","last_clean":"2023-10-20T10:35:33.345534+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:35:33.345534+0000","last_undegraded":"2023-10-20T10:35:33.345534+0000","last_fullsized":"2023-10-20T10:35:33.345534+0000","mapping_epoch":16,"log_start":"134'534632","ondisk_log_start":"134'534632","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'519713","last_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_deep_scrub":"132'519713","last_deep_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_clean_scrub_stamp":"2023-10-20T10:31:19.489275+0000","objects_scrubbed":4836,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":696,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:01:22.001899+0000","scrub_duration":695.57415461999994,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7323996160,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":537890,"num_read_kb":2120948,"num_write":537505,"num_write_kb":2150020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":315,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:35:23.317529+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:35:23.317529+0000","last_peered":"2023-10-20T10:35:23.317529+0000","last_clean":"2023-10-20T10:35:23.317529+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:35:23.317529+0000","last_undegraded":"2023-10-20T10:35:23.317529+0000","last_fullsized":"2023-10-20T10:35:23.317529+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"136'505863","reported_seq":1002699,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:35:33.826686+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:35:33.826686+0000","last_peered":"2023-10-20T10:35:33.826686+0000","last_clean":"2023-10-20T10:35:33.826686+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:35:33.826686+0000","last_undegraded":"2023-10-20T10:35:33.826686+0000","last_fullsized":"2023-10-20T10:35:33.826686+0000","mapping_epoch":16,"log_start":"134'495842","ondisk_log_start":"134'495842","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6793134080,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":499530,"num_read_kb":1967540,"num_write":499119,"num_write_kb":1996476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"135'338","reported_seq":6949,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:35:27.759953+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:35:27.759953+0000","last_peered":"2023-10-20T10:35:27.759953+0000","last_clean":"2023-10-20T10:35:27.759953+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:35:27.759953+0000","last_undegraded":"2023-10-20T10:35:27.759953+0000","last_fullsized":"2023-10-20T10:35:27.759953+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"136'537105","reported_seq":1064495,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:35:33.336314+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:35:33.336314+0000","last_peered":"2023-10-20T10:35:33.336314+0000","last_clean":"2023-10-20T10:35:33.336314+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:35:33.336314+0000","last_undegraded":"2023-10-20T10:35:33.336314+0000","last_fullsized":"2023-10-20T10:35:33.336314+0000","mapping_epoch":16,"log_start":"134'527036","ondisk_log_start":"134'527036","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7214612480,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":530292,"num_read_kb":2089204,"num_write":529867,"num_write_kb":2119468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"135'315","reported_seq":6796,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:35:27.759170+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:35:27.759170+0000","last_peered":"2023-10-20T10:35:27.759170+0000","last_clean":"2023-10-20T10:35:27.759170+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:35:27.759170+0000","last_undegraded":"2023-10-20T10:35:27.759170+0000","last_fullsized":"2023-10-20T10:35:27.759170+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"136'533905","reported_seq":1059612,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:35:33.850263+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:35:33.850263+0000","last_peered":"2023-10-20T10:35:33.850263+0000","last_clean":"2023-10-20T10:35:33.850263+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:35:33.850263+0000","last_undegraded":"2023-10-20T10:35:33.850263+0000","last_fullsized":"2023-10-20T10:35:33.850263+0000","mapping_epoch":16,"log_start":"134'523825","ondisk_log_start":"134'523825","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3033,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 409s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7179419667,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":528518,"num_read_kb":2082436,"num_write":528131,"num_write_kb":2112518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":220,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:35:23.744461+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:35:23.744461+0000","last_peered":"2023-10-20T10:35:23.744461+0000","last_clean":"2023-10-20T10:35:23.744461+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:35:23.744461+0000","last_undegraded":"2023-10-20T10:35:23.744461+0000","last_fullsized":"2023-10-20T10:35:23.744461+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":313,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:34:23.299458+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:34:23.299458+0000","last_peered":"2023-10-20T10:34:23.299458+0000","last_clean":"2023-10-20T10:34:23.299458+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:34:23.299458+0000","last_undegraded":"2023-10-20T10:34:23.299458+0000","last_fullsized":"2023-10-20T10:34:23.299458+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28511162387,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":2096230,"num_read_kb":8260128,"num_write":2094622,"num_write_kb":8378482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17157406720,"data_stored":30540161062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40255,"ondisk_log_size":40255,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53442,"internal_metadata":0},"log_size":30845,"ondisk_log_size":30845,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739354,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9850412,"kb_used_data":8377380,"kb_used_omap":1970,"kb_used_meta":1471053,"kb_avail":84521428,"statfs":{"total":96636764160,"available":86549942272,"internally_reserved":0,"allocated":8578437120,"data_stored":15268913570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2017535,"internal_metadata":1506359041},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.48299999999999998,"15min":0.42499999999999999},"min":{"1min":0.189,"5min":0.17899999999999999,"15min":0.16300000000000001},"max":{"1min":1.0369999999999999,"5min":4.8380000000000001,"15min":6.2430000000000003},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53400000000000003,"15min":0.47299999999999998},"min":{"1min":0.26200000000000001,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.8740000000000001,"5min":4.8259999999999996,"15min":5.048},"last":0.51000000000000001}]},{"osd":1,"last update":"Fri Oct 20 10:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.41699999999999998,"15min":0.498},"min":{"1min":0.19500000000000001,"5min":0.191,"15min":0.186},"max":{"1min":4.476,"5min":4.476,"15min":6.1509999999999998},"last":0.65600000000000003},{"interface":"front","average":{"1min":1.085,"5min":0.61299999999999999,"15min":0.51700000000000002},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":12.16,"5min":12.16,"15min":12.16},"last":0.54200000000000004}]}]},{"osd":1,"up_from":8,"seq":34359739354,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5555480,"kb_used_data":4110480,"kb_used_omap":902,"kb_used_meta":1444089,"kb_avail":88816360,"statfs":{"total":96636764160,"available":90947952640,"internally_reserved":0,"allocated":4209131520,"data_stored":7473270038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":924229,"internal_metadata":1478747579},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,0,6,6,14,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:35:27 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.375,"15min":0.437},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.69599999999999995,"5min":5.0750000000000002,"15min":5.0750000000000002},"last":0.218},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.33500000000000002,"15min":0.41799999999999998},"min":{"1min":0.188,"5min":0.185,"15min":0.185},"max":{"1min":0.64000000000000001,"5min":1.6539999999999999,"15min":5.4640000000000004},"last":0.26200000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:35:27 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.47099999999999997,"15min":0.48099999999999998},"min":{"1min":0.20399999999999999,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":0.629,"5min":4.5899999999999999,"15min":7.6459999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.375,"5min":0.52400000000000002,"15min":0.52900000000000003},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.65700000000000003,"5min":4.625,"15min":7.6820000000000004},"last":0.32700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739356,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6248760,"kb_used_data":4270768,"kb_used_omap":1025,"kb_used_meta":1976958,"kb_avail":88123080,"statfs":{"total":96636764160,"available":90238033920,"internally_reserved":0,"allocated":4373266432,"data_stored":7798384263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1049977,"internal_metadata":2024405639},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.36199999999999999,"15min":0.56899999999999995},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":2.2370000000000001,"5min":2.5059999999999998,"15min":8.0869999999999997},"last":0.186},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.309,"15min":0.45200000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.751,"5min":0.751,"15min":6.875},"last":0.223}]},{"osd":2,"last update":"Fri Oct 20 10:35:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":0.51600000000000001,"15min":0.50700000000000001},"min":{"1min":0.25700000000000001,"5min":0.223,"15min":0.20200000000000001},"max":{"1min":14.673,"5min":14.673,"15min":14.673},"last":0.30099999999999999},{"interface":"front","average":{"1min":1.1990000000000001,"5min":0.57999999999999996,"15min":0.47899999999999998},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":15.686,"5min":15.686,"15min":15.686},"last":0.28399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13921,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4371972096,"data_stored":7798075392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4207620096,"data_stored":7472893971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8577814528,"data_stored":15269191699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:35:38.949 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:35:38.949 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-20T10:35:39.488 INFO:teuthology.orchestra.run.smithi150.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-20T10:35:39.524 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:35:55.242 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:35:55.238+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:35:55.252 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:35:55.246+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:35:59.526 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:36:00.125 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:36:00.126 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:36:00.183 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2608,"stamp":"2023-10-20T10:35:59.406500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28551582211,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2144717,"num_read_kb":8331933,"num_write":2105365,"num_write_kb":8419711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71244,"ondisk_log_size":71244,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22015036,"kb_used_data":16837476,"kb_used_omap":3898,"kb_used_meta":5173637,"kb_avail":261100484,"statfs":{"total":289910292480,"available":267366895616,"internally_reserved":0,"allocated":17241575424,"data_stored":30621296181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3991741,"internal_metadata":5297805123},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,5,10,4,6,20],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23752704,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5896,"num_read_kb":23484,"num_write":5796,"num_write_kb":23184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002042"},"pg_stats":[{"pgid":"2.7","version":"136'10695","reported_seq":15785,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:35:58.477972+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:35:58.477972+0000","last_peered":"2023-10-20T10:35:58.477972+0000","last_clean":"2023-10-20T10:35:58.477972+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:35:58.477972+0000","last_undegraded":"2023-10-20T10:35:58.477972+0000","last_fullsized":"2023-10-20T10:35:58.477972+0000","mapping_epoch":13,"log_start":"46'600","ondisk_log_start":"46'600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"136'21044","reported_seq":32196,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:35:58.478978+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:35:58.478978+0000","last_peered":"2023-10-20T10:35:58.478978+0000","last_clean":"2023-10-20T10:35:58.478978+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:35:58.478978+0000","last_undegraded":"2023-10-20T10:35:58.478978+0000","last_fullsized":"2023-10-20T10:35:58.478978+0000","mapping_epoch":13,"log_start":"118'11000","ondisk_log_start":"118'11000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":221,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:35:23.744496+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:35:23.744496+0000","last_peered":"2023-10-20T10:35:23.744496+0000","last_clean":"2023-10-20T10:35:23.744496+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:35:23.744496+0000","last_undegraded":"2023-10-20T10:35:23.744496+0000","last_fullsized":"2023-10-20T10:35:23.744496+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"136'10655","reported_seq":16521,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:35:58.478448+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:35:58.478448+0000","last_peered":"2023-10-20T10:35:58.478448+0000","last_clean":"2023-10-20T10:35:58.478448+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:35:58.478448+0000","last_undegraded":"2023-10-20T10:35:58.478448+0000","last_fullsized":"2023-10-20T10:35:58.478448+0000","mapping_epoch":13,"log_start":"46'600","ondisk_log_start":"46'600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"136'547218","reported_seq":1084761,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:35:58.472524+0000","last_change":"2023-10-20T10:31:19.489466+0000","last_active":"2023-10-20T10:35:58.472524+0000","last_peered":"2023-10-20T10:35:58.472524+0000","last_clean":"2023-10-20T10:35:58.472524+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:35:58.472524+0000","last_undegraded":"2023-10-20T10:35:58.472524+0000","last_fullsized":"2023-10-20T10:35:58.472524+0000","mapping_epoch":16,"log_start":"134'537132","ondisk_log_start":"134'537132","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'519713","last_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_deep_scrub":"132'519713","last_deep_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_clean_scrub_stamp":"2023-10-20T10:31:19.489275+0000","objects_scrubbed":4836,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":696,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:01:22.001899+0000","scrub_duration":695.57415461999994,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7334203392,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":540399,"num_read_kb":2130940,"num_write":540006,"num_write_kb":2160024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":315,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:35:23.317529+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:35:23.317529+0000","last_peered":"2023-10-20T10:35:23.317529+0000","last_clean":"2023-10-20T10:35:23.317529+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:35:23.317529+0000","last_undegraded":"2023-10-20T10:35:23.317529+0000","last_fullsized":"2023-10-20T10:35:23.317529+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"136'508175","reported_seq":1007441,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:35:59.017223+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:35:59.017223+0000","last_peered":"2023-10-20T10:35:59.017223+0000","last_clean":"2023-10-20T10:35:59.017223+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:35:59.017223+0000","last_undegraded":"2023-10-20T10:35:59.017223+0000","last_fullsized":"2023-10-20T10:35:59.017223+0000","mapping_epoch":16,"log_start":"134'498142","ondisk_log_start":"134'498142","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6802591744,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":501958,"num_read_kb":1977216,"num_write":501431,"num_write_kb":2005724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"135'338","reported_seq":6970,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:35:53.359120+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:35:53.359120+0000","last_peered":"2023-10-20T10:35:53.359120+0000","last_clean":"2023-10-20T10:35:53.359120+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:35:53.359120+0000","last_undegraded":"2023-10-20T10:35:53.359120+0000","last_fullsized":"2023-10-20T10:35:53.359120+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"136'539585","reported_seq":1069448,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:35:58.457483+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:35:58.457483+0000","last_peered":"2023-10-20T10:35:58.457483+0000","last_clean":"2023-10-20T10:35:58.457483+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:35:58.457483+0000","last_undegraded":"2023-10-20T10:35:58.457483+0000","last_fullsized":"2023-10-20T10:35:58.457483+0000","mapping_epoch":16,"log_start":"134'529536","ondisk_log_start":"134'529536","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7224799232,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":532762,"num_read_kb":2099032,"num_write":532347,"num_write_kb":2129388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"135'315","reported_seq":6817,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:35:53.358371+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:35:53.358371+0000","last_peered":"2023-10-20T10:35:53.358371+0000","last_clean":"2023-10-20T10:35:53.358371+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:35:53.358371+0000","last_undegraded":"2023-10-20T10:35:53.358371+0000","last_fullsized":"2023-10-20T10:35:53.358371+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"136'536460","reported_seq":1064592,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:35:59.024555+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:35:59.024555+0000","last_peered":"2023-10-20T10:35:59.024555+0000","last_clean":"2023-10-20T10:35:59.024555+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:35:59.024555+0000","last_undegraded":"2023-10-20T10:35:59.024555+0000","last_fullsized":"2023-10-20T10:35:59.024555+0000","mapping_epoch":16,"log_start":"134'526425","ondisk_log_start":"134'526425","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3194,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 434s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7189393427,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":530927,"num_read_kb":2092024,"num_write":530567,"num_write_kb":2122262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":220,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:35:23.744461+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:35:23.744461+0000","last_peered":"2023-10-20T10:35:23.744461+0000","last_clean":"2023-10-20T10:35:23.744461+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:35:23.744461+0000","last_undegraded":"2023-10-20T10:35:23.744461+0000","last_fullsized":"2023-10-20T10:35:23.744461+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":313,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:34:23.299458+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:34:23.299458+0000","last_peered":"2023-10-20T10:34:23.299458+0000","last_clean":"2023-10-20T10:34:23.299458+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:34:23.299458+0000","last_undegraded":"2023-10-20T10:34:23.299458+0000","last_fullsized":"2023-10-20T10:34:23.299458+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28550987795,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":2106046,"num_read_kb":8299212,"num_write":2104351,"num_write_kb":8417398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17237168128,"data_stored":30619922470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40203,"ondisk_log_size":40203,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53442,"internal_metadata":0},"log_size":30942,"ondisk_log_size":30942,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739359,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10029416,"kb_used_data":8415968,"kb_used_omap":1970,"kb_used_meta":1611469,"kb_avail":84342424,"statfs":{"total":96636764160,"available":86366642176,"internally_reserved":0,"allocated":8617951232,"data_stored":15308423668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2017535,"internal_metadata":1650145025},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.48299999999999998,"15min":0.42499999999999999},"min":{"1min":0.189,"5min":0.17899999999999999,"15min":0.16300000000000001},"max":{"1min":1.0369999999999999,"5min":4.8380000000000001,"15min":6.2430000000000003},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53400000000000003,"15min":0.47299999999999998},"min":{"1min":0.26200000000000001,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.8740000000000001,"5min":4.8259999999999996,"15min":5.048},"last":0.33500000000000002}]},{"osd":1,"last update":"Fri Oct 20 10:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.41699999999999998,"15min":0.498},"min":{"1min":0.19500000000000001,"5min":0.191,"15min":0.186},"max":{"1min":4.476,"5min":4.476,"15min":6.1509999999999998},"last":0.21199999999999999},{"interface":"front","average":{"1min":1.085,"5min":0.61299999999999999,"15min":0.51700000000000002},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":12.16,"5min":12.16,"15min":12.16},"last":0.36799999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739359,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5633384,"kb_used_data":4130080,"kb_used_omap":902,"kb_used_meta":1502393,"kb_avail":88738456,"statfs":{"total":96636764160,"available":90868178944,"internally_reserved":0,"allocated":4229201920,"data_stored":7493336424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":924229,"internal_metadata":1538450875},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,1,3,20],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:35:27 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.375,"15min":0.437},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.69599999999999995,"5min":5.0750000000000002,"15min":5.0750000000000002},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.33500000000000002,"15min":0.41799999999999998},"min":{"1min":0.188,"5min":0.185,"15min":0.185},"max":{"1min":0.64000000000000001,"5min":1.6539999999999999,"15min":5.4640000000000004},"last":0.436}]},{"osd":2,"last update":"Fri Oct 20 10:35:27 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.47099999999999997,"15min":0.48099999999999998},"min":{"1min":0.20399999999999999,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":0.629,"5min":4.5899999999999999,"15min":7.6459999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.375,"5min":0.52400000000000002,"15min":0.52900000000000003},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.65700000000000003,"5min":4.625,"15min":7.6820000000000004},"last":0.49199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739361,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6352236,"kb_used_data":4291428,"kb_used_omap":1025,"kb_used_meta":2059774,"kb_avail":88019604,"statfs":{"total":96636764160,"available":90132074496,"internally_reserved":0,"allocated":4394422272,"data_stored":7819536089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1049977,"internal_metadata":2109209223},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,5,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.36199999999999999,"15min":0.56899999999999995},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":2.2370000000000001,"5min":2.5059999999999998,"15min":8.0869999999999997},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.309,"15min":0.45200000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.751,"5min":0.751,"15min":6.875},"last":0.25700000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:35:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":0.51600000000000001,"15min":0.50700000000000001},"min":{"1min":0.25700000000000001,"5min":0.223,"15min":0.20200000000000001},"max":{"1min":14.673,"5min":14.673,"15min":14.673},"last":0.33600000000000002},{"interface":"front","average":{"1min":1.1990000000000001,"5min":0.57999999999999996,"15min":0.47899999999999998},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":15.686,"5min":15.686,"15min":15.686},"last":0.31900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13921,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4392366080,"data_stored":7818469376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4227059712,"data_stored":7492333587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8617742336,"data_stored":15309119507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:36:00.185 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:36:00.186 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:36:06.569 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:36:06.566+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:36:06.569 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:36:06.566+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:36:20.187 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:36:21.447 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:36:21.447 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:36:21.476 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2619,"stamp":"2023-10-20T10:36:21.410141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28582998531,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2152724,"num_read_kb":8363781,"num_write":2113043,"num_write_kb":8450423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71203,"ondisk_log_size":71203,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22248216,"kb_used_data":16898496,"kb_used_omap":3898,"kb_used_meta":5345797,"kb_avail":260867304,"statfs":{"total":289910292480,"available":267128119296,"internally_reserved":0,"allocated":17304059904,"data_stored":30683780661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3991741,"internal_metadata":5474096963},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,3,2,0,3,7,15,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15491072,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3930,"num_read_kb":15640,"num_write":3780,"num_write_kb":15120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001979"},"pg_stats":[{"pgid":"2.7","version":"136'10734","reported_seq":15836,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:36:18.958108+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:36:18.958108+0000","last_peered":"2023-10-20T10:36:18.958108+0000","last_clean":"2023-10-20T10:36:18.958108+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:36:18.958108+0000","last_undegraded":"2023-10-20T10:36:18.958108+0000","last_fullsized":"2023-10-20T10:36:18.958108+0000","mapping_epoch":13,"log_start":"46'700","ondisk_log_start":"46'700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"136'21121","reported_seq":32298,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:36:18.959520+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:36:18.959520+0000","last_peered":"2023-10-20T10:36:18.959520+0000","last_clean":"2023-10-20T10:36:18.959520+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:36:18.959520+0000","last_undegraded":"2023-10-20T10:36:18.959520+0000","last_fullsized":"2023-10-20T10:36:18.959520+0000","mapping_epoch":13,"log_start":"118'11100","ondisk_log_start":"118'11100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":221,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:35:23.744496+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:35:23.744496+0000","last_peered":"2023-10-20T10:35:23.744496+0000","last_clean":"2023-10-20T10:35:23.744496+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:35:23.744496+0000","last_undegraded":"2023-10-20T10:35:23.744496+0000","last_fullsized":"2023-10-20T10:35:23.744496+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"136'10694","reported_seq":16572,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:36:18.958338+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:36:18.958338+0000","last_peered":"2023-10-20T10:36:18.958338+0000","last_clean":"2023-10-20T10:36:18.958338+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:36:18.958338+0000","last_undegraded":"2023-10-20T10:36:18.958338+0000","last_fullsized":"2023-10-20T10:36:18.958338+0000","mapping_epoch":13,"log_start":"46'600","ondisk_log_start":"46'600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"136'549232","reported_seq":1088850,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:36:18.521639+0000","last_change":"2023-10-20T10:31:19.489466+0000","last_active":"2023-10-20T10:36:18.521639+0000","last_peered":"2023-10-20T10:36:18.521639+0000","last_clean":"2023-10-20T10:36:18.521639+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:36:18.521639+0000","last_undegraded":"2023-10-20T10:36:18.521639+0000","last_fullsized":"2023-10-20T10:36:18.521639+0000","mapping_epoch":16,"log_start":"135'539132","ondisk_log_start":"135'539132","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'519713","last_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_deep_scrub":"132'519713","last_deep_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_clean_scrub_stamp":"2023-10-20T10:31:19.489275+0000","objects_scrubbed":4836,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":696,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:01:22.001899+0000","scrub_duration":695.57415461999994,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7342428160,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":542476,"num_read_kb":2139204,"num_write":542020,"num_write_kb":2168080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":315,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:35:23.317529+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:35:23.317529+0000","last_peered":"2023-10-20T10:35:23.317529+0000","last_clean":"2023-10-20T10:35:23.317529+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:35:23.317529+0000","last_undegraded":"2023-10-20T10:35:23.317529+0000","last_fullsized":"2023-10-20T10:35:23.317529+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"136'509963","reported_seq":1011176,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:36:19.122617+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:36:19.122617+0000","last_peered":"2023-10-20T10:36:19.122617+0000","last_clean":"2023-10-20T10:36:19.122617+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:36:19.122617+0000","last_undegraded":"2023-10-20T10:36:19.122617+0000","last_fullsized":"2023-10-20T10:36:19.122617+0000","mapping_epoch":16,"log_start":"135'499942","ondisk_log_start":"135'499942","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6809907200,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":503905,"num_read_kb":1984956,"num_write":503219,"num_write_kb":2012876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"135'338","reported_seq":6986,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:36:13.844094+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:36:13.844094+0000","last_peered":"2023-10-20T10:36:13.844094+0000","last_clean":"2023-10-20T10:36:13.844094+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:36:13.844094+0000","last_undegraded":"2023-10-20T10:36:13.844094+0000","last_fullsized":"2023-10-20T10:36:13.844094+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"136'541510","reported_seq":1073367,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:36:18.518376+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:36:18.518376+0000","last_peered":"2023-10-20T10:36:18.518376+0000","last_clean":"2023-10-20T10:36:18.518376+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:36:18.518376+0000","last_undegraded":"2023-10-20T10:36:18.518376+0000","last_fullsized":"2023-10-20T10:36:18.518376+0000","mapping_epoch":16,"log_start":"135'531436","ondisk_log_start":"135'531436","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7232712704,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":534755,"num_read_kb":2106964,"num_write":534272,"num_write_kb":2137088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"135'315","reported_seq":6834,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:36:13.842356+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:36:13.842356+0000","last_peered":"2023-10-20T10:36:13.842356+0000","last_clean":"2023-10-20T10:36:13.842356+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:36:13.842356+0000","last_undegraded":"2023-10-20T10:36:13.842356+0000","last_fullsized":"2023-10-20T10:36:13.842356+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"136'538537","reported_seq":1068674,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:36:19.131934+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:36:19.131934+0000","last_peered":"2023-10-20T10:36:19.131934+0000","last_clean":"2023-10-20T10:36:19.131934+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:36:19.131934+0000","last_undegraded":"2023-10-20T10:36:19.131934+0000","last_fullsized":"2023-10-20T10:36:19.131934+0000","mapping_epoch":16,"log_start":"135'528525","ondisk_log_start":"135'528525","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3364,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 454s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7197356051,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":532917,"num_read_kb":2099936,"num_write":532518,"num_write_kb":2130066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":220,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:35:23.744461+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:35:23.744461+0000","last_peered":"2023-10-20T10:35:23.744461+0000","last_clean":"2023-10-20T10:35:23.744461+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:35:23.744461+0000","last_undegraded":"2023-10-20T10:35:23.744461+0000","last_fullsized":"2023-10-20T10:35:23.744461+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":314,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:35:33.320422+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:35:33.320422+0000","last_peered":"2023-10-20T10:35:33.320422+0000","last_clean":"2023-10-20T10:35:33.320422+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:35:33.320422+0000","last_undegraded":"2023-10-20T10:35:33.320422+0000","last_fullsized":"2023-10-20T10:35:33.320422+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28582404115,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":2114053,"num_read_kb":8331060,"num_write":2112029,"num_write_kb":8448110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17299951616,"data_stored":30682705958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40207,"ondisk_log_size":40207,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53442,"internal_metadata":0},"log_size":30897,"ondisk_log_size":30897,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739363,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10141984,"kb_used_data":8446616,"kb_used_omap":1970,"kb_used_meta":1693389,"kb_avail":84229856,"statfs":{"total":96636764160,"available":86251372544,"internally_reserved":0,"allocated":8649334784,"data_stored":15339807220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2017535,"internal_metadata":1734031105},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.48299999999999998,"15min":0.42499999999999999},"min":{"1min":0.189,"5min":0.17899999999999999,"15min":0.16300000000000001},"max":{"1min":1.0369999999999999,"5min":4.8380000000000001,"15min":6.2430000000000003},"last":0.255},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53400000000000003,"15min":0.47299999999999998},"min":{"1min":0.26200000000000001,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":2.8740000000000001,"5min":4.8259999999999996,"15min":5.048},"last":0.26800000000000002}]},{"osd":1,"last update":"Fri Oct 20 10:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.41699999999999998,"15min":0.498},"min":{"1min":0.19500000000000001,"5min":0.191,"15min":0.186},"max":{"1min":4.476,"5min":4.476,"15min":6.1509999999999998},"last":0.32300000000000001},{"interface":"front","average":{"1min":1.085,"5min":0.61299999999999999,"15min":0.51700000000000002},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":12.16,"5min":12.16,"15min":12.16},"last":0.30099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739363,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5702120,"kb_used_data":4144672,"kb_used_omap":902,"kb_used_meta":1556537,"kb_avail":88669720,"statfs":{"total":96636764160,"available":90797793280,"internally_reserved":0,"allocated":4244144128,"data_stored":7508278632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":924229,"internal_metadata":1593894331},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,3,7,15,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:35:27 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.375,"15min":0.437},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.69599999999999995,"5min":5.0750000000000002,"15min":5.0750000000000002},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.33500000000000002,"15min":0.41799999999999998},"min":{"1min":0.188,"5min":0.185,"15min":0.185},"max":{"1min":0.64000000000000001,"5min":1.6539999999999999,"15min":5.4640000000000004},"last":0.21299999999999999}]},{"osd":2,"last update":"Fri Oct 20 10:35:27 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.47099999999999997,"15min":0.48099999999999998},"min":{"1min":0.20399999999999999,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":0.629,"5min":4.5899999999999999,"15min":7.6459999999999999},"last":0.248},{"interface":"front","average":{"1min":0.375,"5min":0.52400000000000002,"15min":0.52900000000000003},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.65700000000000003,"5min":4.625,"15min":7.6820000000000004},"last":0.27600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739365,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6404112,"kb_used_data":4307208,"kb_used_omap":1025,"kb_used_meta":2095870,"kb_avail":87967728,"statfs":{"total":96636764160,"available":90078953472,"internally_reserved":0,"allocated":4410580992,"data_stored":7835694809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1049977,"internal_metadata":2146171527},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:36:16 2023","interfaces":[{"interface":"back","average":{"1min":0.27800000000000002,"5min":0.36099999999999999,"15min":0.55200000000000005},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.41799999999999998,"5min":2.5059999999999998,"15min":8.0869999999999997},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.29099999999999998,"5min":0.30299999999999999,"15min":0.44},"min":{"1min":0.19700000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.39300000000000002,"5min":0.751,"15min":6.875},"last":0.24199999999999999}]},{"osd":2,"last update":"Fri Oct 20 10:36:16 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.51500000000000001,"15min":0.5},"min":{"1min":0.22900000000000001,"5min":0.223,"15min":0.20200000000000001},"max":{"1min":1.0249999999999999,"5min":14.673,"15min":14.673},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.56899999999999995,"15min":0.47999999999999998},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":1.04,"5min":15.686,"15min":15.686},"last":0.312}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13921,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4408492032,"data_stored":7834595328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4242382848,"data_stored":7507656723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8649076736,"data_stored":15340453907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:36:21.478 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:36:21.478 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:36:41.480 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:36:42.032 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:36:42.033 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:36:42.072 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2629,"stamp":"2023-10-20T10:36:41.413487+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28613599747,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2160500,"num_read_kb":8394677,"num_write":2120504,"num_write_kb":8480267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71181,"ondisk_log_size":71181,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22357248,"kb_used_data":16957672,"kb_used_omap":3898,"kb_used_meta":5395653,"kb_avail":260758272,"statfs":{"total":289910292480,"available":267016470528,"internally_reserved":0,"allocated":17364656128,"data_stored":30744376885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3991741,"internal_metadata":5525149507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,16,6,19,28],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15331328,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3872,"num_read_kb":15408,"num_write":3737,"num_write_kb":14948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002031"},"pg_stats":[{"pgid":"2.7","version":"136'10779","reported_seq":15891,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:36:38.788958+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:36:38.788958+0000","last_peered":"2023-10-20T10:36:38.788958+0000","last_clean":"2023-10-20T10:36:38.788958+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:36:38.788958+0000","last_undegraded":"2023-10-20T10:36:38.788958+0000","last_fullsized":"2023-10-20T10:36:38.788958+0000","mapping_epoch":13,"log_start":"46'700","ondisk_log_start":"46'700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"136'21211","reported_seq":32406,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:36:38.817194+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:36:38.817194+0000","last_peered":"2023-10-20T10:36:38.817194+0000","last_clean":"2023-10-20T10:36:38.817194+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:36:38.817194+0000","last_undegraded":"2023-10-20T10:36:38.817194+0000","last_fullsized":"2023-10-20T10:36:38.817194+0000","mapping_epoch":13,"log_start":"118'11200","ondisk_log_start":"118'11200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":221,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:35:23.744496+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:35:23.744496+0000","last_peered":"2023-10-20T10:35:23.744496+0000","last_clean":"2023-10-20T10:35:23.744496+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:35:23.744496+0000","last_undegraded":"2023-10-20T10:35:23.744496+0000","last_fullsized":"2023-10-20T10:35:23.744496+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"136'10739","reported_seq":16626,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:36:38.794464+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:36:38.794464+0000","last_peered":"2023-10-20T10:36:38.794464+0000","last_clean":"2023-10-20T10:36:38.794464+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:36:38.794464+0000","last_undegraded":"2023-10-20T10:36:38.794464+0000","last_fullsized":"2023-10-20T10:36:38.794464+0000","mapping_epoch":13,"log_start":"46'700","ondisk_log_start":"46'700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"136'551163","reported_seq":1092817,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:36:38.663979+0000","last_change":"2023-10-20T10:31:19.489466+0000","last_active":"2023-10-20T10:36:38.663979+0000","last_peered":"2023-10-20T10:36:38.663979+0000","last_clean":"2023-10-20T10:36:38.663979+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:36:38.663979+0000","last_undegraded":"2023-10-20T10:36:38.663979+0000","last_fullsized":"2023-10-20T10:36:38.663979+0000","mapping_epoch":16,"log_start":"135'541132","ondisk_log_start":"135'541132","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'519713","last_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_deep_scrub":"132'519713","last_deep_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_clean_scrub_stamp":"2023-10-20T10:31:19.489275+0000","objects_scrubbed":4836,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":696,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:01:22.001899+0000","scrub_duration":695.57415461999994,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7350353920,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":544516,"num_read_kb":2147316,"num_write":543951,"num_write_kb":2175804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":315,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:35:23.317529+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:35:23.317529+0000","last_peered":"2023-10-20T10:35:23.317529+0000","last_clean":"2023-10-20T10:35:23.317529+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:35:23.317529+0000","last_undegraded":"2023-10-20T10:35:23.317529+0000","last_fullsized":"2023-10-20T10:35:23.317529+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"136'511660","reported_seq":1014698,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:36:39.262636+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:36:39.262636+0000","last_peered":"2023-10-20T10:36:39.262636+0000","last_clean":"2023-10-20T10:36:39.262636+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:36:39.262636+0000","last_undegraded":"2023-10-20T10:36:39.262636+0000","last_fullsized":"2023-10-20T10:36:39.262636+0000","mapping_epoch":16,"log_start":"135'501642","ondisk_log_start":"135'501642","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6816845824,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":505732,"num_read_kb":1992216,"num_write":504916,"num_write_kb":2019664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"135'338","reported_seq":7002,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:36:34.395169+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:36:34.395169+0000","last_peered":"2023-10-20T10:36:34.395169+0000","last_clean":"2023-10-20T10:36:34.395169+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:36:34.395169+0000","last_undegraded":"2023-10-20T10:36:34.395169+0000","last_fullsized":"2023-10-20T10:36:34.395169+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"136'543422","reported_seq":1077251,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:36:38.640241+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:36:38.640241+0000","last_peered":"2023-10-20T10:36:38.640241+0000","last_clean":"2023-10-20T10:36:38.640241+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:36:38.640241+0000","last_undegraded":"2023-10-20T10:36:38.640241+0000","last_fullsized":"2023-10-20T10:36:38.640241+0000","mapping_epoch":16,"log_start":"135'533336","ondisk_log_start":"135'533336","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7240560640,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":536728,"num_read_kb":2114820,"num_write":536184,"num_write_kb":2144736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"135'315","reported_seq":6850,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:36:34.355874+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:36:34.355874+0000","last_peered":"2023-10-20T10:36:34.355874+0000","last_clean":"2023-10-20T10:36:34.355874+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:36:34.355874+0000","last_undegraded":"2023-10-20T10:36:34.355874+0000","last_fullsized":"2023-10-20T10:36:34.355874+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"136'540595","reported_seq":1072679,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:36:39.270290+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:36:39.270290+0000","last_peered":"2023-10-20T10:36:39.270290+0000","last_clean":"2023-10-20T10:36:39.270290+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:36:39.270290+0000","last_undegraded":"2023-10-20T10:36:39.270290+0000","last_fullsized":"2023-10-20T10:36:39.270290+0000","mapping_epoch":16,"log_start":"135'530525","ondisk_log_start":"135'530525","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3545,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 474s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7205244947,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":534853,"num_read_kb":2107604,"num_write":534439,"num_write_kb":2137750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":220,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-20T10:35:23.744461+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:35:23.744461+0000","last_peered":"2023-10-20T10:35:23.744461+0000","last_clean":"2023-10-20T10:35:23.744461+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:35:23.744461+0000","last_undegraded":"2023-10-20T10:35:23.744461+0000","last_fullsized":"2023-10-20T10:35:23.744461+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":314,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:35:33.320422+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:35:33.320422+0000","last_peered":"2023-10-20T10:35:33.320422+0000","last_clean":"2023-10-20T10:35:33.320422+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:35:33.320422+0000","last_undegraded":"2023-10-20T10:35:33.320422+0000","last_fullsized":"2023-10-20T10:35:33.320422+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28613005331,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":2121829,"num_read_kb":8361956,"num_write":2119490,"num_write_kb":8477954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17360805888,"data_stored":30743560230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40205,"ondisk_log_size":40205,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53442,"internal_metadata":0},"log_size":30877,"ondisk_log_size":30877,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739367,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10187368,"kb_used_data":8476064,"kb_used_omap":1970,"kb_used_meta":1709325,"kb_avail":84184472,"statfs":{"total":96636764160,"available":86204899328,"internally_reserved":0,"allocated":8679489536,"data_stored":15369961972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2017535,"internal_metadata":1750349569},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,4,3,6,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:36:24 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47399999999999998,"15min":0.438},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.16300000000000001},"max":{"1min":4.8129999999999997,"5min":4.8380000000000001,"15min":6.2430000000000003},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.54900000000000004,"15min":0.48799999999999999},"min":{"1min":0.23499999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":4.8540000000000001,"5min":4.8540000000000001,"15min":5.048},"last":0.312}]},{"osd":1,"last update":"Fri Oct 20 10:36:24 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.42999999999999999,"15min":0.501},"min":{"1min":0.20899999999999999,"5min":0.191,"15min":0.186},"max":{"1min":0.98299999999999998,"5min":4.476,"15min":6.1509999999999998},"last":0.214},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66200000000000003,"15min":0.53100000000000003},"min":{"1min":0.25600000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.3369999999999997,"5min":12.16,"15min":12.16},"last":0.26200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739367,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5734548,"kb_used_data":4159052,"kb_used_omap":902,"kb_used_meta":1574585,"kb_avail":88637292,"statfs":{"total":96636764160,"available":90764587008,"internally_reserved":0,"allocated":4258869248,"data_stored":7523003752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":924229,"internal_metadata":1612375483},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,0,12,18],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:36:27 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.40500000000000003,"15min":0.44600000000000001},"min":{"1min":0.184,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":4.0149999999999997,"5min":5.0750000000000002,"15min":5.0750000000000002},"last":0.183},{"interface":"front","average":{"1min":0.31,"5min":0.32300000000000001,"15min":0.42199999999999999},"min":{"1min":0.20799999999999999,"5min":0.185,"15min":0.185},"max":{"1min":0.502,"5min":1.6539999999999999,"15min":5.4640000000000004},"last":0.30299999999999999}]},{"osd":2,"last update":"Fri Oct 20 10:36:27 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.46400000000000002,"15min":0.48399999999999999},"min":{"1min":0.223,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":0.50900000000000001,"5min":4.5899999999999999,"15min":7.6459999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.51600000000000001,"15min":0.53100000000000003},"min":{"1min":0.27600000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.61299999999999999,"5min":4.625,"15min":7.6820000000000004},"last":0.377}]}]},{"osd":0,"up_from":8,"seq":34359739369,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6435332,"kb_used_data":4322556,"kb_used_omap":1025,"kb_used_meta":2111742,"kb_avail":87936508,"statfs":{"total":96636764160,"available":90046984192,"internally_reserved":0,"allocated":4426297344,"data_stored":7851411161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1049977,"internal_metadata":2162424455},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,11,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:36:16 2023","interfaces":[{"interface":"back","average":{"1min":0.27800000000000002,"5min":0.36099999999999999,"15min":0.55200000000000005},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.41799999999999998,"5min":2.5059999999999998,"15min":8.0869999999999997},"last":0.314},{"interface":"front","average":{"1min":0.29099999999999998,"5min":0.30299999999999999,"15min":0.44},"min":{"1min":0.19700000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.39300000000000002,"5min":0.751,"15min":6.875},"last":0.28100000000000003}]},{"osd":2,"last update":"Fri Oct 20 10:36:16 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.51500000000000001,"15min":0.5},"min":{"1min":0.22900000000000001,"5min":0.223,"15min":0.20200000000000001},"max":{"1min":1.0249999999999999,"5min":14.673,"15min":14.673},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.56899999999999995,"15min":0.47999999999999998},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":1.04,"5min":15.686,"15min":15.686},"last":0.39200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13921,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4424241152,"data_stored":7850344448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4257202176,"data_stored":7522476051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8679362560,"data_stored":15370739731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:36:42.075 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:36:42.076 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:36:53.042 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:36:53.033+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T10:36:53.043 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:36:53.033+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:36:53.050 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:36:53.041+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T10:36:53.050 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:36:53.045+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T10:36:53.051 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:36:53.049+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:36:53.051 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:36:53.049+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:36:53.055 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:36:53.053+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2750 2023-10-20T10:37:02.077 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:37:02.531 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:37:02.532 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:37:02.569 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2640,"stamp":"2023-10-20T10:37:01.416925+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28644418051,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2168393,"num_read_kb":8426061,"num_write":2128022,"num_write_kb":8510339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71280,"ondisk_log_size":71280,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21961272,"kb_used_data":17018144,"kb_used_omap":3898,"kb_used_meta":4939205,"kb_avail":261154248,"statfs":{"total":289910292480,"available":267421949952,"internally_reserved":0,"allocated":17426579456,"data_stored":30806288042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3991741,"internal_metadata":5057746755},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,3,2,12,20,10,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15310848,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3860,"num_read_kb":15344,"num_write":3727,"num_write_kb":14908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002101"},"pg_stats":[{"pgid":"2.7","version":"137'10824","reported_seq":15949,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:36:58.886678+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:36:58.886678+0000","last_peered":"2023-10-20T10:36:58.886678+0000","last_clean":"2023-10-20T10:36:58.886678+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:36:58.886678+0000","last_undegraded":"2023-10-20T10:36:58.886678+0000","last_fullsized":"2023-10-20T10:36:58.886678+0000","mapping_epoch":13,"log_start":"48'800","ondisk_log_start":"48'800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"137'21301","reported_seq":32521,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:36:58.907361+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:36:58.907361+0000","last_peered":"2023-10-20T10:36:58.907361+0000","last_clean":"2023-10-20T10:36:58.907361+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:36:58.907361+0000","last_undegraded":"2023-10-20T10:36:58.907361+0000","last_fullsized":"2023-10-20T10:36:58.907361+0000","mapping_epoch":13,"log_start":"118'11300","ondisk_log_start":"118'11300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":222,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:35:33.318608+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:35:33.318608+0000","last_peered":"2023-10-20T10:35:33.318608+0000","last_clean":"2023-10-20T10:35:33.318608+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:35:33.318608+0000","last_undegraded":"2023-10-20T10:35:33.318608+0000","last_fullsized":"2023-10-20T10:35:33.318608+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"137'10784","reported_seq":16684,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:36:58.889501+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:36:58.889501+0000","last_peered":"2023-10-20T10:36:58.889501+0000","last_clean":"2023-10-20T10:36:58.889501+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:36:58.889501+0000","last_undegraded":"2023-10-20T10:36:58.889501+0000","last_fullsized":"2023-10-20T10:36:58.889501+0000","mapping_epoch":13,"log_start":"46'700","ondisk_log_start":"46'700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"137'553082","reported_seq":1096818,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:36:58.746997+0000","last_change":"2023-10-20T10:31:19.489466+0000","last_active":"2023-10-20T10:36:58.746997+0000","last_peered":"2023-10-20T10:36:58.746997+0000","last_clean":"2023-10-20T10:36:58.746997+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:36:58.746997+0000","last_undegraded":"2023-10-20T10:36:58.746997+0000","last_fullsized":"2023-10-20T10:36:58.746997+0000","mapping_epoch":16,"log_start":"135'543032","ondisk_log_start":"135'543032","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'519713","last_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_deep_scrub":"132'519713","last_deep_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_clean_scrub_stamp":"2023-10-20T10:31:19.489275+0000","objects_scrubbed":4836,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":696,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:01:22.001899+0000","scrub_duration":695.57415461999994,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7358251008,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":546591,"num_read_kb":2155564,"num_write":545870,"num_write_kb":2183480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":316,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:35:33.305706+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:35:33.305706+0000","last_peered":"2023-10-20T10:35:33.305706+0000","last_clean":"2023-10-20T10:35:33.305706+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:35:33.305706+0000","last_undegraded":"2023-10-20T10:35:33.305706+0000","last_fullsized":"2023-10-20T10:35:33.305706+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"137'513418","reported_seq":1018348,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:36:59.311431+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:36:59.311431+0000","last_peered":"2023-10-20T10:36:59.311431+0000","last_clean":"2023-10-20T10:36:59.311431+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:36:59.311431+0000","last_undegraded":"2023-10-20T10:36:59.311431+0000","last_fullsized":"2023-10-20T10:36:59.311431+0000","mapping_epoch":16,"log_start":"135'503342","ondisk_log_start":"135'503342","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6824022016,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":507620,"num_read_kb":1999724,"num_write":506674,"num_write_kb":2026696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"135'338","reported_seq":7019,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:36:54.808908+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:36:54.808908+0000","last_peered":"2023-10-20T10:36:54.808908+0000","last_clean":"2023-10-20T10:36:54.808908+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:36:54.808908+0000","last_undegraded":"2023-10-20T10:36:54.808908+0000","last_fullsized":"2023-10-20T10:36:54.808908+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"137'545334","reported_seq":1081210,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:36:58.732047+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:36:58.732047+0000","last_peered":"2023-10-20T10:36:58.732047+0000","last_clean":"2023-10-20T10:36:58.732047+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:36:58.732047+0000","last_undegraded":"2023-10-20T10:36:58.732047+0000","last_fullsized":"2023-10-20T10:36:58.732047+0000","mapping_epoch":16,"log_start":"135'535236","ondisk_log_start":"135'535236","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7248388096,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":538776,"num_read_kb":2122964,"num_write":538096,"num_write_kb":2152384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"135'315","reported_seq":6867,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:36:54.802692+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:36:54.802692+0000","last_peered":"2023-10-20T10:36:54.802692+0000","last_clean":"2023-10-20T10:36:54.802692+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:36:54.802692+0000","last_undegraded":"2023-10-20T10:36:54.802692+0000","last_fullsized":"2023-10-20T10:36:54.802692+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"137'542626","reported_seq":1076606,"reported_epoch":137,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:36:59.322437+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:36:59.322437+0000","last_peered":"2023-10-20T10:36:59.322437+0000","last_clean":"2023-10-20T10:36:59.322437+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:36:59.322437+0000","last_undegraded":"2023-10-20T10:36:59.322437+0000","last_fullsized":"2023-10-20T10:36:59.322437+0000","mapping_epoch":16,"log_start":"135'532526","ondisk_log_start":"135'532526","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3679,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 494s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7213162515,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":536735,"num_read_kb":2115088,"num_write":536368,"num_write_kb":2145466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":221,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:35:33.318604+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:35:33.318604+0000","last_peered":"2023-10-20T10:35:33.318604+0000","last_clean":"2023-10-20T10:35:33.318604+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:35:33.318604+0000","last_undegraded":"2023-10-20T10:35:33.318604+0000","last_fullsized":"2023-10-20T10:35:33.318604+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":315,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:36:04.038434+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:36:04.038434+0000","last_peered":"2023-10-20T10:36:04.038434+0000","last_clean":"2023-10-20T10:36:04.038434+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:36:04.038434+0000","last_undegraded":"2023-10-20T10:36:04.038434+0000","last_fullsized":"2023-10-20T10:36:04.038434+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28643823635,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":2129722,"num_read_kb":8393340,"num_write":2127008,"num_write_kb":8508026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17422614528,"data_stored":30805368870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40324,"ondisk_log_size":40324,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53442,"internal_metadata":0},"log_size":30857,"ondisk_log_size":30857,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739371,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10320272,"kb_used_data":8506248,"kb_used_omap":1970,"kb_used_meta":1812045,"kb_avail":84051568,"statfs":{"total":96636764160,"available":86068805632,"internally_reserved":0,"allocated":8710397952,"data_stored":15400866331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2017535,"internal_metadata":1855534849},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,2,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:36:24 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47399999999999998,"15min":0.438},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.16300000000000001},"max":{"1min":4.8129999999999997,"5min":4.8380000000000001,"15min":6.2430000000000003},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.54900000000000004,"15min":0.48799999999999999},"min":{"1min":0.23499999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":4.8540000000000001,"5min":4.8540000000000001,"15min":5.048},"last":0.26900000000000002}]},{"osd":1,"last update":"Fri Oct 20 10:36:24 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.42999999999999999,"15min":0.501},"min":{"1min":0.20899999999999999,"5min":0.191,"15min":0.186},"max":{"1min":0.98299999999999998,"5min":4.476,"15min":6.1509999999999998},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66200000000000003,"15min":0.53100000000000003},"min":{"1min":0.25600000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.3369999999999997,"5min":12.16,"15min":12.16},"last":0.25}]}]},{"osd":1,"up_from":8,"seq":34359739371,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5785548,"kb_used_data":4173956,"kb_used_omap":902,"kb_used_meta":1610681,"kb_avail":88586292,"statfs":{"total":96636764160,"available":90712363008,"internally_reserved":0,"allocated":4274130944,"data_stored":7538261391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":924229,"internal_metadata":1649337787},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,0,4,9,10,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:36:27 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.40500000000000003,"15min":0.44600000000000001},"min":{"1min":0.184,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":4.0149999999999997,"5min":5.0750000000000002,"15min":5.0750000000000002},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.31,"5min":0.32300000000000001,"15min":0.42199999999999999},"min":{"1min":0.20799999999999999,"5min":0.185,"15min":0.185},"max":{"1min":0.502,"5min":1.6539999999999999,"15min":5.4640000000000004},"last":0.45400000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:36:27 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.46400000000000002,"15min":0.48399999999999999},"min":{"1min":0.223,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":0.50900000000000001,"5min":4.5899999999999999,"15min":7.6459999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.51600000000000001,"15min":0.53100000000000003},"min":{"1min":0.27600000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.61299999999999999,"5min":4.625,"15min":7.6820000000000004},"last":0.47699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739373,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5855452,"kb_used_data":4337940,"kb_used_omap":1025,"kb_used_meta":1516478,"kb_avail":88516388,"statfs":{"total":96636764160,"available":90640781312,"internally_reserved":0,"allocated":4442050560,"data_stored":7867160320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1049977,"internal_metadata":1552874119},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:36:16 2023","interfaces":[{"interface":"back","average":{"1min":0.27800000000000002,"5min":0.36099999999999999,"15min":0.55200000000000005},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.41799999999999998,"5min":2.5059999999999998,"15min":8.0869999999999997},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.29099999999999998,"5min":0.30299999999999999,"15min":0.44},"min":{"1min":0.19700000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.39300000000000002,"5min":0.751,"15min":6.875},"last":0.47799999999999998}]},{"osd":2,"last update":"Fri Oct 20 10:36:16 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.51500000000000001,"15min":0.5},"min":{"1min":0.22900000000000001,"5min":0.223,"15min":0.20200000000000001},"max":{"1min":1.0249999999999999,"5min":14.673,"15min":14.673},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.56899999999999995,"15min":0.47999999999999998},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":1.04,"5min":15.686,"15min":15.686},"last":0.46300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13921,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4439924736,"data_stored":7866028032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4272300032,"data_stored":7537573907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8710389760,"data_stored":15401766931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:37:02.571 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:37:02.572 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:37:12.097 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:37:12.097+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:37:12.127 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:37:12.129+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:37:12.128 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:37:12.129+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:37:22.573 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:37:22.976 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:37:22.977 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:37:23.002 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2650,"stamp":"2023-10-20T10:37:21.420031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28676403715,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2176689,"num_read_kb":8459085,"num_write":2135829,"num_write_kb":8541567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71166,"ondisk_log_size":71166,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22196608,"kb_used_data":17081320,"kb_used_omap":3898,"kb_used_meta":5111365,"kb_avail":260918912,"statfs":{"total":289910292480,"available":267180965888,"internally_reserved":0,"allocated":17491271680,"data_stored":30870980266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3991741,"internal_metadata":5234038595},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,5,10,6,5,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16338944,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4152,"num_read_kb":16512,"num_write":3981,"num_write_kb":15924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001859"},"pg_stats":[{"pgid":"2.7","version":"137'10870","reported_seq":16007,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:37:18.022441+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:37:18.022441+0000","last_peered":"2023-10-20T10:37:18.022441+0000","last_clean":"2023-10-20T10:37:18.022441+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:37:18.022441+0000","last_undegraded":"2023-10-20T10:37:18.022441+0000","last_fullsized":"2023-10-20T10:37:18.022441+0000","mapping_epoch":13,"log_start":"48'800","ondisk_log_start":"48'800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"137'21392","reported_seq":32636,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:37:18.056809+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:37:18.056809+0000","last_peered":"2023-10-20T10:37:18.056809+0000","last_clean":"2023-10-20T10:37:18.056809+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:37:18.056809+0000","last_undegraded":"2023-10-20T10:37:18.056809+0000","last_fullsized":"2023-10-20T10:37:18.056809+0000","mapping_epoch":13,"log_start":"118'11300","ondisk_log_start":"118'11300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":222,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:35:33.318608+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:35:33.318608+0000","last_peered":"2023-10-20T10:35:33.318608+0000","last_clean":"2023-10-20T10:35:33.318608+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:35:33.318608+0000","last_undegraded":"2023-10-20T10:35:33.318608+0000","last_fullsized":"2023-10-20T10:35:33.318608+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"137'10830","reported_seq":16742,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:37:18.028178+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:37:18.028178+0000","last_peered":"2023-10-20T10:37:18.028178+0000","last_clean":"2023-10-20T10:37:18.028178+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:37:18.028178+0000","last_undegraded":"2023-10-20T10:37:18.028178+0000","last_fullsized":"2023-10-20T10:37:18.028178+0000","mapping_epoch":13,"log_start":"48'800","ondisk_log_start":"48'800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"137'555146","reported_seq":1101084,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:37:18.833574+0000","last_change":"2023-10-20T10:31:19.489466+0000","last_active":"2023-10-20T10:37:18.833574+0000","last_peered":"2023-10-20T10:37:18.833574+0000","last_clean":"2023-10-20T10:37:18.833574+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:37:18.833574+0000","last_undegraded":"2023-10-20T10:37:18.833574+0000","last_fullsized":"2023-10-20T10:37:18.833574+0000","mapping_epoch":16,"log_start":"136'545132","ondisk_log_start":"136'545132","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'519713","last_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_deep_scrub":"132'519713","last_deep_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_clean_scrub_stamp":"2023-10-20T10:31:19.489275+0000","objects_scrubbed":4836,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":696,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:01:22.001899+0000","scrub_duration":695.57415461999994,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7366705152,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":548796,"num_read_kb":2164352,"num_write":547934,"num_write_kb":2191736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":316,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:35:33.305706+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:35:33.305706+0000","last_peered":"2023-10-20T10:35:33.305706+0000","last_clean":"2023-10-20T10:35:33.305706+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:35:33.305706+0000","last_undegraded":"2023-10-20T10:35:33.305706+0000","last_fullsized":"2023-10-20T10:35:33.305706+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"137'515264","reported_seq":1022149,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:37:19.412129+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:37:19.412129+0000","last_peered":"2023-10-20T10:37:19.412129+0000","last_clean":"2023-10-20T10:37:19.412129+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:37:19.412129+0000","last_undegraded":"2023-10-20T10:37:19.412129+0000","last_fullsized":"2023-10-20T10:37:19.412129+0000","mapping_epoch":16,"log_start":"135'505242","ondisk_log_start":"135'505242","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6831583232,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":509575,"num_read_kb":2007520,"num_write":508520,"num_write_kb":2034080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"135'338","reported_seq":7036,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:37:15.279449+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:37:15.279449+0000","last_peered":"2023-10-20T10:37:15.279449+0000","last_clean":"2023-10-20T10:37:15.279449+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:37:15.279449+0000","last_undegraded":"2023-10-20T10:37:15.279449+0000","last_fullsized":"2023-10-20T10:37:15.279449+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"137'547266","reported_seq":1085240,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:37:18.822898+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:37:18.822898+0000","last_peered":"2023-10-20T10:37:18.822898+0000","last_clean":"2023-10-20T10:37:18.822898+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:37:18.822898+0000","last_undegraded":"2023-10-20T10:37:18.822898+0000","last_fullsized":"2023-10-20T10:37:18.822898+0000","mapping_epoch":16,"log_start":"136'537236","ondisk_log_start":"136'537236","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7256305664,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":540875,"num_read_kb":2131324,"num_write":540028,"num_write_kb":2160112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"135'315","reported_seq":6883,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:37:15.278353+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:37:15.278353+0000","last_peered":"2023-10-20T10:37:15.278353+0000","last_clean":"2023-10-20T10:37:15.278353+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:37:15.278353+0000","last_undegraded":"2023-10-20T10:37:15.278353+0000","last_fullsized":"2023-10-20T10:37:15.278353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"137'544687","reported_seq":1080718,"reported_epoch":137,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:37:19.423681+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:37:19.423681+0000","last_peered":"2023-10-20T10:37:19.423681+0000","last_clean":"2023-10-20T10:37:19.423681+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:37:19.423681+0000","last_undegraded":"2023-10-20T10:37:19.423681+0000","last_fullsized":"2023-10-20T10:37:19.423681+0000","mapping_epoch":16,"log_start":"136'534626","ondisk_log_start":"136'534626","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3811,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 514s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7221215251,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":538772,"num_read_kb":2123168,"num_write":538333,"num_write_kb":2153326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":221,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:35:33.318604+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:35:33.318604+0000","last_peered":"2023-10-20T10:35:33.318604+0000","last_clean":"2023-10-20T10:35:33.318604+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:35:33.318604+0000","last_undegraded":"2023-10-20T10:35:33.318604+0000","last_fullsized":"2023-10-20T10:35:33.318604+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":315,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:36:04.038434+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:36:04.038434+0000","last_peered":"2023-10-20T10:36:04.038434+0000","last_clean":"2023-10-20T10:36:04.038434+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:36:04.038434+0000","last_undegraded":"2023-10-20T10:36:04.038434+0000","last_fullsized":"2023-10-20T10:36:04.038434+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28675809299,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":2138018,"num_read_kb":8426364,"num_write":2134815,"num_write_kb":8539254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17486782464,"data_stored":30869536806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40127,"ondisk_log_size":40127,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53442,"internal_metadata":0},"log_size":30940,"ondisk_log_size":30940,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739375,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10439088,"kb_used_data":8538984,"kb_used_omap":1970,"kb_used_meta":1898125,"kb_avail":83932752,"statfs":{"total":96636764160,"available":85947138048,"internally_reserved":0,"allocated":8743919616,"data_stored":15434387995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2017535,"internal_metadata":1943680769},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,2,1,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:36:24 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47399999999999998,"15min":0.438},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.16300000000000001},"max":{"1min":4.8129999999999997,"5min":4.8380000000000001,"15min":6.2430000000000003},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.54900000000000004,"15min":0.48799999999999999},"min":{"1min":0.23499999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":4.8540000000000001,"5min":4.8540000000000001,"15min":5.048},"last":0.21299999999999999}]},{"osd":1,"last update":"Fri Oct 20 10:36:24 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.42999999999999999,"15min":0.501},"min":{"1min":0.20899999999999999,"5min":0.191,"15min":0.186},"max":{"1min":0.98299999999999998,"5min":4.476,"15min":6.1509999999999998},"last":0.245},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66200000000000003,"15min":0.53100000000000003},"min":{"1min":0.25600000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":6.3369999999999997,"5min":12.16,"15min":12.16},"last":0.28199999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739375,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5836860,"kb_used_data":4189172,"kb_used_omap":902,"kb_used_meta":1646777,"kb_avail":88534980,"statfs":{"total":96636764160,"available":90659819520,"internally_reserved":0,"allocated":4289712128,"data_stored":7553842575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":924229,"internal_metadata":1686300091},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,3,2,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:36:27 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.40500000000000003,"15min":0.44600000000000001},"min":{"1min":0.184,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":4.0149999999999997,"5min":5.0750000000000002,"15min":5.0750000000000002},"last":0.16300000000000001},{"interface":"front","average":{"1min":0.31,"5min":0.32300000000000001,"15min":0.42199999999999999},"min":{"1min":0.20799999999999999,"5min":0.185,"15min":0.185},"max":{"1min":0.502,"5min":1.6539999999999999,"15min":5.4640000000000004},"last":0.41199999999999998}]},{"osd":2,"last update":"Fri Oct 20 10:36:27 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.46400000000000002,"15min":0.48399999999999999},"min":{"1min":0.223,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":0.50900000000000001,"5min":4.5899999999999999,"15min":7.6459999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.51600000000000001,"15min":0.53100000000000003},"min":{"1min":0.27600000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.61299999999999999,"5min":4.625,"15min":7.6820000000000004},"last":0.47499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739377,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5920660,"kb_used_data":4353164,"kb_used_omap":1025,"kb_used_meta":1566462,"kb_avail":88451180,"statfs":{"total":96636764160,"available":90574008320,"internally_reserved":0,"allocated":4457639936,"data_stored":7882749696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1049977,"internal_metadata":1604057735},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,6,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:37:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.41299999999999998,"15min":0.57499999999999996},"min":{"1min":0.20000000000000001,"5min":0.158,"15min":0.158},"max":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":8.0869999999999997},"last":3.0830000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.35599999999999998,"15min":0.45200000000000001},"min":{"1min":0.23499999999999999,"5min":0.188,"15min":0.188},"max":{"1min":6.1449999999999996,"5min":6.1449999999999996,"15min":6.875},"last":0.26900000000000002}]},{"osd":2,"last update":"Fri Oct 20 10:37:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.54800000000000004,"15min":0.51700000000000002},"min":{"1min":0.25700000000000001,"5min":0.223,"15min":0.20200000000000001},"max":{"1min":6.1989999999999998,"5min":14.673,"15min":14.673},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.56899999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.193,"15min":0.193},"max":{"1min":1.8819999999999999,"5min":15.686,"15min":15.686},"last":0.224}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13921,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4456292352,"data_stored":7882395648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4287905792,"data_stored":7553179667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8742584320,"data_stored":15433961491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:37:23.004 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:37:23.005 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:37:43.006 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:37:43.578 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:37:43.579 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:37:43.609 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2661,"stamp":"2023-10-20T10:37:43.423808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28707488259,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2184532,"num_read_kb":8490293,"num_write":2143422,"num_write_kb":8571939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71293,"ondisk_log_size":71293,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22282604,"kb_used_data":17148564,"kb_used_omap":3898,"kb_used_meta":5130117,"kb_avail":260832916,"statfs":{"total":289910292480,"available":267092905984,"internally_reserved":0,"allocated":17560129536,"data_stored":30939838122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3991741,"internal_metadata":5253240643},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,5,52,7,4,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15650816,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3881,"num_read_kb":15448,"num_write":3819,"num_write_kb":15276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002034"},"pg_stats":[{"pgid":"2.7","version":"137'10903","reported_seq":16052,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:37:38.307458+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:37:38.307458+0000","last_peered":"2023-10-20T10:37:38.307458+0000","last_clean":"2023-10-20T10:37:38.307458+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:37:38.307458+0000","last_undegraded":"2023-10-20T10:37:38.307458+0000","last_fullsized":"2023-10-20T10:37:38.307458+0000","mapping_epoch":13,"log_start":"48'900","ondisk_log_start":"48'900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"137'21458","reported_seq":32726,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:37:38.313581+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:37:38.313581+0000","last_peered":"2023-10-20T10:37:38.313581+0000","last_clean":"2023-10-20T10:37:38.313581+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:37:38.313581+0000","last_undegraded":"2023-10-20T10:37:38.313581+0000","last_fullsized":"2023-10-20T10:37:38.313581+0000","mapping_epoch":13,"log_start":"118'11400","ondisk_log_start":"118'11400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":223,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:36:43.319130+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:36:43.319130+0000","last_peered":"2023-10-20T10:36:43.319130+0000","last_clean":"2023-10-20T10:36:43.319130+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:36:43.319130+0000","last_undegraded":"2023-10-20T10:36:43.319130+0000","last_fullsized":"2023-10-20T10:36:43.319130+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"137'10863","reported_seq":16788,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:37:38.304396+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:37:38.304396+0000","last_peered":"2023-10-20T10:37:38.304396+0000","last_clean":"2023-10-20T10:37:38.304396+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:37:38.304396+0000","last_undegraded":"2023-10-20T10:37:38.304396+0000","last_fullsized":"2023-10-20T10:37:38.304396+0000","mapping_epoch":13,"log_start":"48'800","ondisk_log_start":"48'800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"137'557122","reported_seq":1105010,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:37:38.869625+0000","last_change":"2023-10-20T10:31:19.489466+0000","last_active":"2023-10-20T10:37:38.869625+0000","last_peered":"2023-10-20T10:37:38.869625+0000","last_clean":"2023-10-20T10:37:38.869625+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:37:38.869625+0000","last_undegraded":"2023-10-20T10:37:38.869625+0000","last_fullsized":"2023-10-20T10:37:38.869625+0000","mapping_epoch":16,"log_start":"136'547032","ondisk_log_start":"136'547032","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'519713","last_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_deep_scrub":"132'519713","last_deep_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_clean_scrub_stamp":"2023-10-20T10:31:19.489275+0000","objects_scrubbed":4836,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":696,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:01:22.001899+0000","scrub_duration":695.57415461999994,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7374778368,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":550743,"num_read_kb":2172108,"num_write":549910,"num_write_kb":2199640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":317,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:36:43.316807+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:36:43.316807+0000","last_peered":"2023-10-20T10:36:43.316807+0000","last_clean":"2023-10-20T10:36:43.316807+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:36:43.316807+0000","last_undegraded":"2023-10-20T10:36:43.316807+0000","last_fullsized":"2023-10-20T10:36:43.316807+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"137'517035","reported_seq":1025844,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:37:39.520937+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:37:39.520937+0000","last_peered":"2023-10-20T10:37:39.520937+0000","last_clean":"2023-10-20T10:37:39.520937+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:37:39.520937+0000","last_undegraded":"2023-10-20T10:37:39.520937+0000","last_fullsized":"2023-10-20T10:37:39.520937+0000","mapping_epoch":16,"log_start":"136'506942","ondisk_log_start":"136'506942","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6838824960,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":511499,"num_read_kb":2015172,"num_write":510291,"num_write_kb":2041164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"135'338","reported_seq":7056,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:37:40.878853+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:37:40.878853+0000","last_peered":"2023-10-20T10:37:40.878853+0000","last_clean":"2023-10-20T10:37:40.878853+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:37:40.878853+0000","last_undegraded":"2023-10-20T10:37:40.878853+0000","last_fullsized":"2023-10-20T10:37:40.878853+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"137'549181","reported_seq":1089218,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:37:38.863399+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:37:38.863399+0000","last_peered":"2023-10-20T10:37:38.863399+0000","last_clean":"2023-10-20T10:37:38.863399+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:37:38.863399+0000","last_undegraded":"2023-10-20T10:37:38.863399+0000","last_fullsized":"2023-10-20T10:37:38.863399+0000","mapping_epoch":16,"log_start":"136'539136","ondisk_log_start":"136'539136","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7264165888,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":542936,"num_read_kb":2139524,"num_write":541943,"num_write_kb":2167772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"135'315","reported_seq":6903,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:37:40.879130+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:37:40.879130+0000","last_peered":"2023-10-20T10:37:40.879130+0000","last_clean":"2023-10-20T10:37:40.879130+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:37:40.879130+0000","last_undegraded":"2023-10-20T10:37:40.879130+0000","last_fullsized":"2023-10-20T10:37:40.879130+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"137'546720","reported_seq":1084677,"reported_epoch":137,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:37:39.469861+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:37:39.469861+0000","last_peered":"2023-10-20T10:37:39.469861+0000","last_clean":"2023-10-20T10:37:39.469861+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:37:39.469861+0000","last_undegraded":"2023-10-20T10:37:39.469861+0000","last_fullsized":"2023-10-20T10:37:39.469861+0000","mapping_epoch":16,"log_start":"136'536626","ondisk_log_start":"136'536626","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":3960,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 534s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7229124627,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":540683,"num_read_kb":2130768,"num_write":540264,"num_write_kb":2161050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":222,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:36:43.319192+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:36:43.319192+0000","last_peered":"2023-10-20T10:36:43.319192+0000","last_clean":"2023-10-20T10:36:43.319192+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:36:43.319192+0000","last_undegraded":"2023-10-20T10:36:43.319192+0000","last_fullsized":"2023-10-20T10:36:43.319192+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":315,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-20T10:36:04.038434+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:36:04.038434+0000","last_peered":"2023-10-20T10:36:04.038434+0000","last_clean":"2023-10-20T10:36:04.038434+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:36:04.038434+0000","last_undegraded":"2023-10-20T10:36:04.038434+0000","last_fullsized":"2023-10-20T10:36:04.038434+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28706893843,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":2145861,"num_read_kb":8457572,"num_write":2142408,"num_write_kb":8569626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17556369408,"data_stored":30939123750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40322,"ondisk_log_size":40322,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53442,"internal_metadata":0},"log_size":30872,"ondisk_log_size":30872,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739380,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10442224,"kb_used_data":8575272,"kb_used_omap":1970,"kb_used_meta":1864973,"kb_avail":83929616,"statfs":{"total":96636764160,"available":85943926784,"internally_reserved":0,"allocated":8781078528,"data_stored":15471546907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2017535,"internal_metadata":1909733121},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,1,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:37:29 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.41199999999999998,"15min":0.44500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17100000000000001,"15min":0.16300000000000001},"max":{"1min":4.1539999999999999,"5min":4.8129999999999997,"15min":6.2430000000000003},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.45800000000000002,"15min":0.49099999999999999},"min":{"1min":0.21299999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.69099999999999995,"5min":4.8540000000000001,"15min":5.048},"last":0.29899999999999999}]},{"osd":1,"last update":"Fri Oct 20 10:37:29 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.435,"15min":0.499},"min":{"1min":0.20200000000000001,"5min":0.191,"15min":0.186},"max":{"1min":1.974,"5min":4.476,"15min":6.1509999999999998},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.66000000000000003,"15min":0.53300000000000003},"min":{"1min":0.24399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.75600000000000001,"5min":12.16,"15min":12.16},"last":0.33700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739379,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5887596,"kb_used_data":4203812,"kb_used_omap":902,"kb_used_meta":1682873,"kb_avail":88484244,"statfs":{"total":96636764160,"available":90607865856,"internally_reserved":0,"allocated":4304703488,"data_stored":7568833935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":924229,"internal_metadata":1723262395},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,48,6,4,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:37:27 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.41199999999999998,"15min":0.443},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91100000000000003,"5min":5.0750000000000002,"15min":5.0750000000000002},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.40000000000000002,"15min":0.438},"min":{"1min":0.254,"5min":0.188,"15min":0.185},"max":{"1min":4.7670000000000003,"5min":4.7670000000000003,"15min":5.4640000000000004},"last":0.23200000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:37:27 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.43099999999999999,"15min":0.48899999999999999},"min":{"1min":0.247,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":0.61899999999999999,"5min":4.5899999999999999,"15min":7.6459999999999999},"last":0.443},{"interface":"front","average":{"1min":0.434,"5min":0.48599999999999999,"15min":0.53800000000000003},"min":{"1min":0.318,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.67600000000000005,"5min":4.625,"15min":7.6820000000000004},"last":0.46600000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739381,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5952784,"kb_used_data":4369480,"kb_used_omap":1025,"kb_used_meta":1582270,"kb_avail":88419056,"statfs":{"total":96636764160,"available":90541113344,"internally_reserved":0,"allocated":4474347520,"data_stored":7899457280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1049977,"internal_metadata":1620245127},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:37:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.41299999999999998,"15min":0.57499999999999996},"min":{"1min":0.20000000000000001,"5min":0.158,"15min":0.158},"max":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":8.0869999999999997},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.35599999999999998,"15min":0.45200000000000001},"min":{"1min":0.23499999999999999,"5min":0.188,"15min":0.188},"max":{"1min":6.1449999999999996,"5min":6.1449999999999996,"15min":6.875},"last":1.1080000000000001}]},{"osd":2,"last update":"Fri Oct 20 10:37:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.54800000000000004,"15min":0.51700000000000002},"min":{"1min":0.25700000000000001,"5min":0.223,"15min":0.20200000000000001},"max":{"1min":6.1989999999999998,"5min":14.673,"15min":14.673},"last":1.3009999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.56899999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.193,"15min":0.193},"max":{"1min":1.8819999999999999,"5min":15.686,"15min":15.686},"last":1.272}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13921,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4472238080,"data_stored":7898341376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4303077376,"data_stored":7568351251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8781053952,"data_stored":15472431123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:37:43.611 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:37:43.611 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-20T10:37:44.131 INFO:teuthology.orchestra.run.smithi150.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-20T10:37:44.160 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:37:49.824 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:37:49.824+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T10:37:49.824 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:37:49.824+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:37:49.824 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:37:49.824+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T10:37:49.825 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:37:49.824+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T10:37:49.829 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:37:49.828+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:37:49.829 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:37:49.828+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:38:04.161 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:38:04.590 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:38:04.591 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:38:04.629 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2672,"stamp":"2023-10-20T10:38:03.427997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28738830851,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2192681,"num_read_kb":8522789,"num_write":2151069,"num_write_kb":8602527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71176,"ondisk_log_size":71176,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22501624,"kb_used_data":17211228,"kb_used_omap":3898,"kb_used_meta":5286469,"kb_avail":260613896,"statfs":{"total":289910292480,"available":266868629504,"internally_reserved":0,"allocated":17624297472,"data_stored":31003993758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3991741,"internal_metadata":5413345091},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,5,11,13,20,13,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15798272,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4013,"num_read_kb":16004,"num_write":3858,"num_write_kb":15432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.082406"},"pg_stats":[{"pgid":"2.7","version":"138'10958","reported_seq":16120,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:37:56.241980+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:37:56.241980+0000","last_peered":"2023-10-20T10:37:56.241980+0000","last_clean":"2023-10-20T10:37:56.241980+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:37:56.241980+0000","last_undegraded":"2023-10-20T10:37:56.241980+0000","last_fullsized":"2023-10-20T10:37:56.241980+0000","mapping_epoch":13,"log_start":"48'900","ondisk_log_start":"48'900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"138'21563","reported_seq":32856,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:37:56.242542+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:37:56.242542+0000","last_peered":"2023-10-20T10:37:56.242542+0000","last_clean":"2023-10-20T10:37:56.242542+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:37:56.242542+0000","last_undegraded":"2023-10-20T10:37:56.242542+0000","last_fullsized":"2023-10-20T10:37:56.242542+0000","mapping_epoch":13,"log_start":"118'11500","ondisk_log_start":"118'11500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":224,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:37:29.467068+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:37:29.467068+0000","last_peered":"2023-10-20T10:37:29.467068+0000","last_clean":"2023-10-20T10:37:29.467068+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:37:29.467068+0000","last_undegraded":"2023-10-20T10:37:29.467068+0000","last_fullsized":"2023-10-20T10:37:29.467068+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"138'10917","reported_seq":16855,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:37:56.242310+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:37:56.242310+0000","last_peered":"2023-10-20T10:37:56.242310+0000","last_clean":"2023-10-20T10:37:56.242310+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:37:56.242310+0000","last_undegraded":"2023-10-20T10:37:56.242310+0000","last_fullsized":"2023-10-20T10:37:56.242310+0000","mapping_epoch":13,"log_start":"48'900","ondisk_log_start":"48'900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"138'559108","reported_seq":1109077,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:37:58.965671+0000","last_change":"2023-10-20T10:31:19.489466+0000","last_active":"2023-10-20T10:37:58.965671+0000","last_peered":"2023-10-20T10:37:58.965671+0000","last_clean":"2023-10-20T10:37:58.965671+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:37:58.965671+0000","last_undegraded":"2023-10-20T10:37:58.965671+0000","last_fullsized":"2023-10-20T10:37:58.965671+0000","mapping_epoch":16,"log_start":"136'549032","ondisk_log_start":"136'549032","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'519713","last_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_deep_scrub":"132'519713","last_deep_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_clean_scrub_stamp":"2023-10-20T10:31:19.489275+0000","objects_scrubbed":4836,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":696,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:01:22.001899+0000","scrub_duration":695.57415461999994,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7382863872,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":552823,"num_read_kb":2180384,"num_write":551896,"num_write_kb":2207584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":318,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:37:28.863588+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:37:28.863588+0000","last_peered":"2023-10-20T10:37:28.863588+0000","last_clean":"2023-10-20T10:37:28.863588+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:37:28.863588+0000","last_undegraded":"2023-10-20T10:37:28.863588+0000","last_fullsized":"2023-10-20T10:37:28.863588+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"138'518881","reported_seq":1029622,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:37:59.648824+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:37:59.648824+0000","last_peered":"2023-10-20T10:37:59.648824+0000","last_clean":"2023-10-20T10:37:59.648824+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:37:59.648824+0000","last_undegraded":"2023-10-20T10:37:59.648824+0000","last_fullsized":"2023-10-20T10:37:59.648824+0000","mapping_epoch":16,"log_start":"136'508842","ondisk_log_start":"136'508842","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6846423040,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":513434,"num_read_kb":2022888,"num_write":512137,"num_write_kb":2048548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"135'338","reported_seq":7073,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:38:01.358385+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:38:01.358385+0000","last_peered":"2023-10-20T10:38:01.358385+0000","last_clean":"2023-10-20T10:38:01.358385+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:38:01.358385+0000","last_undegraded":"2023-10-20T10:38:01.358385+0000","last_fullsized":"2023-10-20T10:38:01.358385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"138'551147","reported_seq":1093235,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:37:58.976239+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:37:58.976239+0000","last_peered":"2023-10-20T10:37:58.976239+0000","last_clean":"2023-10-20T10:37:58.976239+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:37:58.976239+0000","last_undegraded":"2023-10-20T10:37:58.976239+0000","last_fullsized":"2023-10-20T10:37:58.976239+0000","mapping_epoch":16,"log_start":"136'541136","ondisk_log_start":"136'541136","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7272243200,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":544983,"num_read_kb":2147704,"num_write":543909,"num_write_kb":2175636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"135'315","reported_seq":6920,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:38:01.364081+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:38:01.364081+0000","last_peered":"2023-10-20T10:38:01.364081+0000","last_clean":"2023-10-20T10:38:01.364081+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:38:01.364081+0000","last_undegraded":"2023-10-20T10:38:01.364081+0000","last_fullsized":"2023-10-20T10:38:01.364081+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"138'548691","reported_seq":1088746,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:37:59.655999+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:37:59.655999+0000","last_peered":"2023-10-20T10:37:59.655999+0000","last_clean":"2023-10-20T10:37:59.655999+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:37:59.655999+0000","last_undegraded":"2023-10-20T10:37:59.655999+0000","last_fullsized":"2023-10-20T10:37:59.655999+0000","mapping_epoch":16,"log_start":"136'538626","ondisk_log_start":"136'538626","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":4110,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 554s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7236706323,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":542770,"num_read_kb":2139092,"num_write":542113,"num_write_kb":2168446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":223,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:37:29.466999+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:37:29.466999+0000","last_peered":"2023-10-20T10:37:29.466999+0000","last_clean":"2023-10-20T10:37:29.466999+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:37:29.466999+0000","last_undegraded":"2023-10-20T10:37:29.466999+0000","last_fullsized":"2023-10-20T10:37:29.466999+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":316,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:36:43.330313+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:36:43.330313+0000","last_peered":"2023-10-20T10:36:43.330313+0000","last_clean":"2023-10-20T10:36:43.330313+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:36:43.330313+0000","last_undegraded":"2023-10-20T10:36:43.330313+0000","last_fullsized":"2023-10-20T10:36:43.330313+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28738236435,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":2154010,"num_read_kb":8490068,"num_write":2150055,"num_write_kb":8600214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17619730432,"data_stored":31002484774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40191,"ondisk_log_size":40191,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53442,"internal_metadata":0},"log_size":30886,"ondisk_log_size":30886,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739384,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10560792,"kb_used_data":8607756,"kb_used_omap":1970,"kb_used_meta":1951053,"kb_avail":83811048,"statfs":{"total":96636764160,"available":85822513152,"internally_reserved":0,"allocated":8814342144,"data_stored":15504806423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2017535,"internal_metadata":1997879041},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:37:29 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.41199999999999998,"15min":0.44500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17100000000000001,"15min":0.16300000000000001},"max":{"1min":4.1539999999999999,"5min":4.8129999999999997,"15min":6.2430000000000003},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.45800000000000002,"15min":0.49099999999999999},"min":{"1min":0.21299999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.69099999999999995,"5min":4.8540000000000001,"15min":5.048},"last":0.32500000000000001}]},{"osd":1,"last update":"Fri Oct 20 10:37:29 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.435,"15min":0.499},"min":{"1min":0.20200000000000001,"5min":0.191,"15min":0.186},"max":{"1min":1.974,"5min":4.476,"15min":6.1509999999999998},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.66000000000000003,"15min":0.53300000000000003},"min":{"1min":0.24399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.75600000000000001,"5min":12.16,"15min":12.16},"last":0.47699999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739383,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5918300,"kb_used_data":4218388,"kb_used_omap":902,"kb_used_meta":1699001,"kb_avail":88453540,"statfs":{"total":96636764160,"available":90576424960,"internally_reserved":0,"allocated":4319629312,"data_stored":7583755659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":924229,"internal_metadata":1739777467},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,6,5,7,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:37:27 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.41199999999999998,"15min":0.443},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91100000000000003,"5min":5.0750000000000002,"15min":5.0750000000000002},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.40000000000000002,"15min":0.438},"min":{"1min":0.254,"5min":0.188,"15min":0.185},"max":{"1min":4.7670000000000003,"5min":4.7670000000000003,"15min":5.4640000000000004},"last":0.215}]},{"osd":2,"last update":"Fri Oct 20 10:37:27 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.43099999999999999,"15min":0.48899999999999999},"min":{"1min":0.247,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":0.61899999999999999,"5min":4.5899999999999999,"15min":7.6459999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.434,"5min":0.48599999999999999,"15min":0.53800000000000003},"min":{"1min":0.318,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.67600000000000005,"5min":4.625,"15min":7.6820000000000004},"last":0.36399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739385,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6022532,"kb_used_data":4385084,"kb_used_omap":1025,"kb_used_meta":1636414,"kb_avail":88349308,"statfs":{"total":96636764160,"available":90469691392,"internally_reserved":0,"allocated":4490326016,"data_stored":7915431676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1049977,"internal_metadata":1675688583},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,2,4,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:37:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.41299999999999998,"15min":0.57499999999999996},"min":{"1min":0.20000000000000001,"5min":0.158,"15min":0.158},"max":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":8.0869999999999997},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.35599999999999998,"15min":0.45200000000000001},"min":{"1min":0.23499999999999999,"5min":0.188,"15min":0.188},"max":{"1min":6.1449999999999996,"5min":6.1449999999999996,"15min":6.875},"last":0.23799999999999999}]},{"osd":2,"last update":"Fri Oct 20 10:37:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.54800000000000004,"15min":0.51700000000000002},"min":{"1min":0.25700000000000001,"5min":0.223,"15min":0.20200000000000001},"max":{"1min":6.1989999999999998,"5min":14.673,"15min":14.673},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.56899999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.193,"15min":0.193},"max":{"1min":1.8819999999999999,"5min":15.686,"15min":15.686},"last":0.34499999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13921,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4488417280,"data_stored":7914520576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4318212096,"data_stored":7583485971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8813101056,"data_stored":15504478227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:38:04.630 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:38:04.631 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:38:16.308 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:38:16.308+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T10:38:16.310 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:38:16.308+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:38:16.310 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:38:16.308+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:38:16.332 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:38:16.332+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2750 2023-10-20T10:38:24.632 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:38:25.162 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:38:25.163 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:38:25.189 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2682,"stamp":"2023-10-20T10:38:23.431381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28770619907,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2200656,"num_read_kb":8554577,"num_write":2158816,"num_write_kb":8633515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71146,"ondisk_log_size":71146,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22303620,"kb_used_data":17272040,"kb_used_omap":3898,"kb_used_meta":5027653,"kb_avail":260811900,"statfs":{"total":289910292480,"available":267071385600,"internally_reserved":0,"allocated":17686568960,"data_stored":31066265246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3991741,"internal_metadata":5148317507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,3,6,11,21,12,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16097280,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4028,"num_read_kb":16048,"num_write":3918,"num_write_kb":15672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001949"},"pg_stats":[{"pgid":"2.7","version":"138'10992","reported_seq":16166,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:38:17.410762+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:38:17.410762+0000","last_peered":"2023-10-20T10:38:17.410762+0000","last_clean":"2023-10-20T10:38:17.410762+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:38:17.410762+0000","last_undegraded":"2023-10-20T10:38:17.410762+0000","last_fullsized":"2023-10-20T10:38:17.410762+0000","mapping_epoch":13,"log_start":"48'900","ondisk_log_start":"48'900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"138'21630","reported_seq":32947,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:38:17.431649+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:38:17.431649+0000","last_peered":"2023-10-20T10:38:17.431649+0000","last_clean":"2023-10-20T10:38:17.431649+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:38:17.431649+0000","last_undegraded":"2023-10-20T10:38:17.431649+0000","last_fullsized":"2023-10-20T10:38:17.431649+0000","mapping_epoch":13,"log_start":"118'11600","ondisk_log_start":"118'11600","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":224,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:37:29.467068+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:37:29.467068+0000","last_peered":"2023-10-20T10:37:29.467068+0000","last_clean":"2023-10-20T10:37:29.467068+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:37:29.467068+0000","last_undegraded":"2023-10-20T10:37:29.467068+0000","last_fullsized":"2023-10-20T10:37:29.467068+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"138'10950","reported_seq":16900,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:38:17.411857+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:38:17.411857+0000","last_peered":"2023-10-20T10:38:17.411857+0000","last_clean":"2023-10-20T10:38:17.411857+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:38:17.411857+0000","last_undegraded":"2023-10-20T10:38:17.411857+0000","last_fullsized":"2023-10-20T10:38:17.411857+0000","mapping_epoch":13,"log_start":"48'900","ondisk_log_start":"48'900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"138'561144","reported_seq":1113220,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:38:19.069710+0000","last_change":"2023-10-20T10:31:19.489466+0000","last_active":"2023-10-20T10:38:19.069710+0000","last_peered":"2023-10-20T10:38:19.069710+0000","last_clean":"2023-10-20T10:38:19.069710+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:38:19.069710+0000","last_undegraded":"2023-10-20T10:38:19.069710+0000","last_fullsized":"2023-10-20T10:38:19.069710+0000","mapping_epoch":16,"log_start":"136'551132","ondisk_log_start":"136'551132","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'519713","last_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_deep_scrub":"132'519713","last_deep_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_clean_scrub_stamp":"2023-10-20T10:31:19.489275+0000","objects_scrubbed":4836,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":696,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:01:22.001899+0000","scrub_duration":695.57415461999994,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7391244288,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":554930,"num_read_kb":2188784,"num_write":553932,"num_write_kb":2215728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":318,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:37:28.863588+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:37:28.863588+0000","last_peered":"2023-10-20T10:37:28.863588+0000","last_clean":"2023-10-20T10:37:28.863588+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:37:28.863588+0000","last_undegraded":"2023-10-20T10:37:28.863588+0000","last_fullsized":"2023-10-20T10:37:28.863588+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"138'520688","reported_seq":1033347,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:38:19.714771+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:38:19.714771+0000","last_peered":"2023-10-20T10:38:19.714771+0000","last_clean":"2023-10-20T10:38:19.714771+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:38:19.714771+0000","last_undegraded":"2023-10-20T10:38:19.714771+0000","last_fullsized":"2023-10-20T10:38:19.714771+0000","mapping_epoch":16,"log_start":"136'510642","ondisk_log_start":"136'510642","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6853824512,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":515348,"num_read_kb":2030520,"num_write":513944,"num_write_kb":2055776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"135'338","reported_seq":7089,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:38:21.841941+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:38:21.841941+0000","last_peered":"2023-10-20T10:38:21.841941+0000","last_clean":"2023-10-20T10:38:21.841941+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:38:21.841941+0000","last_undegraded":"2023-10-20T10:38:21.841941+0000","last_fullsized":"2023-10-20T10:38:21.841941+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"138'553144","reported_seq":1097230,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:38:19.058185+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:38:19.058185+0000","last_peered":"2023-10-20T10:38:19.058185+0000","last_clean":"2023-10-20T10:38:19.058185+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:38:19.058185+0000","last_undegraded":"2023-10-20T10:38:19.058185+0000","last_fullsized":"2023-10-20T10:38:19.058185+0000","mapping_epoch":16,"log_start":"136'543136","ondisk_log_start":"136'543136","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7280431104,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":546981,"num_read_kb":2155660,"num_write":545906,"num_write_kb":2183624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"135'315","reported_seq":6937,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:38:21.838892+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:38:21.838892+0000","last_peered":"2023-10-20T10:38:21.838892+0000","last_clean":"2023-10-20T10:38:21.838892+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:38:21.838892+0000","last_undegraded":"2023-10-20T10:38:21.838892+0000","last_fullsized":"2023-10-20T10:38:21.838892+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"138'550687","reported_seq":1092709,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:38:19.716556+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:38:19.716556+0000","last_peered":"2023-10-20T10:38:19.716556+0000","last_clean":"2023-10-20T10:38:19.716556+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:38:19.716556+0000","last_undegraded":"2023-10-20T10:38:19.716556+0000","last_fullsized":"2023-10-20T10:38:19.716556+0000","mapping_epoch":16,"log_start":"136'540626","ondisk_log_start":"136'540626","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":4225,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 574s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7244525587,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":544726,"num_read_kb":2146892,"num_write":544020,"num_write_kb":2176074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":223,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:37:29.466999+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:37:29.466999+0000","last_peered":"2023-10-20T10:37:29.466999+0000","last_clean":"2023-10-20T10:37:29.466999+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:37:29.466999+0000","last_undegraded":"2023-10-20T10:37:29.466999+0000","last_fullsized":"2023-10-20T10:37:29.466999+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":317,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:37:53.361632+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:37:53.361632+0000","last_peered":"2023-10-20T10:37:53.361632+0000","last_clean":"2023-10-20T10:37:53.361632+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:37:53.361632+0000","last_undegraded":"2023-10-20T10:37:53.361632+0000","last_fullsized":"2023-10-20T10:37:53.361632+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28770025491,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":2161985,"num_read_kb":8521856,"num_write":2157802,"num_write_kb":8631202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17683189760,"data_stored":31065944102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40127,"ondisk_log_size":40127,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53442,"internal_metadata":0},"log_size":30920,"ondisk_log_size":30920,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739388,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10279448,"kb_used_data":8637456,"kb_used_omap":1970,"kb_used_meta":1640013,"kb_avail":84092392,"statfs":{"total":96636764160,"available":86110609408,"internally_reserved":0,"allocated":8844754944,"data_stored":15535219223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2017535,"internal_metadata":1679374081},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,8,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:37:29 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.41199999999999998,"15min":0.44500000000000001},"min":{"1min":0.17799999999999999,"5min":0.17100000000000001,"15min":0.16300000000000001},"max":{"1min":4.1539999999999999,"5min":4.8129999999999997,"15min":6.2430000000000003},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.45800000000000002,"15min":0.49099999999999999},"min":{"1min":0.21299999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.69099999999999995,"5min":4.8540000000000001,"15min":5.048},"last":0.27800000000000002}]},{"osd":1,"last update":"Fri Oct 20 10:37:29 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.435,"15min":0.499},"min":{"1min":0.20200000000000001,"5min":0.191,"15min":0.186},"max":{"1min":1.974,"5min":4.476,"15min":6.1509999999999998},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.66000000000000003,"15min":0.53300000000000003},"min":{"1min":0.24399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.75600000000000001,"5min":12.16,"15min":12.16},"last":0.31}]}]},{"osd":1,"up_from":8,"seq":34359739387,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5949416,"kb_used_data":4233376,"kb_used_omap":902,"kb_used_meta":1715129,"kb_avail":88422424,"statfs":{"total":96636764160,"available":90544562176,"internally_reserved":0,"allocated":4334977024,"data_stored":7599103371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":924229,"internal_metadata":1756292539},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,2,12,12,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:37:27 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.41199999999999998,"15min":0.443},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91100000000000003,"5min":5.0750000000000002,"15min":5.0750000000000002},"last":0.254},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.40000000000000002,"15min":0.438},"min":{"1min":0.254,"5min":0.188,"15min":0.185},"max":{"1min":4.7670000000000003,"5min":4.7670000000000003,"15min":5.4640000000000004},"last":0.40100000000000002}]},{"osd":2,"last update":"Fri Oct 20 10:37:27 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.43099999999999999,"15min":0.48899999999999999},"min":{"1min":0.247,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":0.61899999999999999,"5min":4.5899999999999999,"15min":7.6459999999999999},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.434,"5min":0.48599999999999999,"15min":0.53800000000000003},"min":{"1min":0.318,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.67600000000000005,"5min":4.625,"15min":7.6820000000000004},"last":0.28000000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739389,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6074756,"kb_used_data":4401208,"kb_used_omap":1025,"kb_used_meta":1672510,"kb_avail":88297084,"statfs":{"total":96636764160,"available":90416214016,"internally_reserved":0,"allocated":4506836992,"data_stored":7931942652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1049977,"internal_metadata":1712650887},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:37:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.41299999999999998,"15min":0.57499999999999996},"min":{"1min":0.20000000000000001,"5min":0.158,"15min":0.158},"max":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":8.0869999999999997},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.35599999999999998,"15min":0.45200000000000001},"min":{"1min":0.23499999999999999,"5min":0.188,"15min":0.188},"max":{"1min":6.1449999999999996,"5min":6.1449999999999996,"15min":6.875},"last":0.373}]},{"osd":2,"last update":"Fri Oct 20 10:37:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.54800000000000004,"15min":0.51700000000000002},"min":{"1min":0.25700000000000001,"5min":0.223,"15min":0.20200000000000001},"max":{"1min":6.1989999999999998,"5min":14.673,"15min":14.673},"last":6.2389999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.56899999999999995,"15min":0.48099999999999998},"min":{"1min":0.20200000000000001,"5min":0.193,"15min":0.193},"max":{"1min":1.8819999999999999,"5min":15.686,"15min":15.686},"last":6.1779999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13921,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4504948736,"data_stored":7931052032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4333416448,"data_stored":7598690323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8844824576,"data_stored":15536201747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:38:25.191 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:38:25.192 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:38:25.449 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:38:25.447+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:38:25.460 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:38:25.459+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:38:25.475 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:38:25.471+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:38:45.193 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:38:45.704 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:38:45.876 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:38:45.877 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2693,"stamp":"2023-10-20T10:38:45.439588+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28811026947,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2210739,"num_read_kb":8594781,"num_write":2168658,"num_write_kb":8672883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71036,"ondisk_log_size":71036,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22605504,"kb_used_data":17344680,"kb_used_omap":3898,"kb_used_meta":5256901,"kb_avail":260510016,"statfs":{"total":289910292480,"available":266762256384,"internally_reserved":0,"allocated":17760952320,"data_stored":31140648606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3991741,"internal_metadata":5383067459},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,4,13,13,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24309760,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6052,"num_read_kb":24124,"num_write":5919,"num_write_kb":23676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002059"},"pg_stats":[{"pgid":"2.7","version":"138'11047","reported_seq":16237,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:38:43.256994+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:38:43.256994+0000","last_peered":"2023-10-20T10:38:43.256994+0000","last_clean":"2023-10-20T10:38:43.256994+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:38:43.256994+0000","last_undegraded":"2023-10-20T10:38:43.256994+0000","last_fullsized":"2023-10-20T10:38:43.256994+0000","mapping_epoch":13,"log_start":"50'1000","ondisk_log_start":"50'1000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"138'21739","reported_seq":33087,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:38:43.286008+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:38:43.286008+0000","last_peered":"2023-10-20T10:38:43.286008+0000","last_clean":"2023-10-20T10:38:43.286008+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:38:43.286008+0000","last_undegraded":"2023-10-20T10:38:43.286008+0000","last_fullsized":"2023-10-20T10:38:43.286008+0000","mapping_epoch":13,"log_start":"118'11700","ondisk_log_start":"118'11700","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":224,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:37:29.467068+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:37:29.467068+0000","last_peered":"2023-10-20T10:37:29.467068+0000","last_clean":"2023-10-20T10:37:29.467068+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:37:29.467068+0000","last_undegraded":"2023-10-20T10:37:29.467068+0000","last_fullsized":"2023-10-20T10:37:29.467068+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"138'11005","reported_seq":16970,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:38:43.263937+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:38:43.263937+0000","last_peered":"2023-10-20T10:38:43.263937+0000","last_clean":"2023-10-20T10:38:43.263937+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:38:43.263937+0000","last_undegraded":"2023-10-20T10:38:43.263937+0000","last_fullsized":"2023-10-20T10:38:43.263937+0000","mapping_epoch":13,"log_start":"50'1000","ondisk_log_start":"50'1000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"138'563721","reported_seq":1118380,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:38:44.139861+0000","last_change":"2023-10-20T10:31:19.489466+0000","last_active":"2023-10-20T10:38:44.139861+0000","last_peered":"2023-10-20T10:38:44.139861+0000","last_clean":"2023-10-20T10:38:44.139861+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:38:44.139861+0000","last_undegraded":"2023-10-20T10:38:44.139861+0000","last_fullsized":"2023-10-20T10:38:44.139861+0000","mapping_epoch":16,"log_start":"137'553632","ondisk_log_start":"137'553632","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'519713","last_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_deep_scrub":"132'519713","last_deep_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_clean_scrub_stamp":"2023-10-20T10:31:19.489275+0000","objects_scrubbed":4836,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":696,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:01:22.001899+0000","scrub_duration":695.57415461999994,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7401762816,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":557512,"num_read_kb":2199076,"num_write":556509,"num_write_kb":2226036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":318,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:37:28.863588+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:37:28.863588+0000","last_peered":"2023-10-20T10:37:28.863588+0000","last_clean":"2023-10-20T10:37:28.863588+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:37:28.863588+0000","last_undegraded":"2023-10-20T10:37:28.863588+0000","last_fullsized":"2023-10-20T10:37:28.863588+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"138'522945","reported_seq":1038032,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:38:44.841319+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:38:44.841319+0000","last_peered":"2023-10-20T10:38:44.841319+0000","last_clean":"2023-10-20T10:38:44.841319+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:38:44.841319+0000","last_undegraded":"2023-10-20T10:38:44.841319+0000","last_fullsized":"2023-10-20T10:38:44.841319+0000","mapping_epoch":16,"log_start":"137'512942","ondisk_log_start":"137'512942","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6863118336,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":517776,"num_read_kb":2040204,"num_write":516201,"num_write_kb":2064804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"135'338","reported_seq":7105,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:38:42.322358+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:38:42.322358+0000","last_peered":"2023-10-20T10:38:42.322358+0000","last_clean":"2023-10-20T10:38:42.322358+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:38:42.322358+0000","last_undegraded":"2023-10-20T10:38:42.322358+0000","last_fullsized":"2023-10-20T10:38:42.322358+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"138'555637","reported_seq":1102274,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:38:44.141853+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:38:44.141853+0000","last_peered":"2023-10-20T10:38:44.141853+0000","last_clean":"2023-10-20T10:38:44.141853+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:38:44.141853+0000","last_undegraded":"2023-10-20T10:38:44.141853+0000","last_fullsized":"2023-10-20T10:38:44.141853+0000","mapping_epoch":16,"log_start":"137'545636","ondisk_log_start":"137'545636","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7290687488,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":549531,"num_read_kb":2165820,"num_write":548399,"num_write_kb":2193596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"135'315","reported_seq":6953,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:38:42.341621+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:38:42.341621+0000","last_peered":"2023-10-20T10:38:42.341621+0000","last_clean":"2023-10-20T10:38:42.341621+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:38:42.341621+0000","last_undegraded":"2023-10-20T10:38:42.341621+0000","last_fullsized":"2023-10-20T10:38:42.341621+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"138'553331","reported_seq":1097891,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:38:44.877334+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:38:44.877334+0000","last_peered":"2023-10-20T10:38:44.877334+0000","last_clean":"2023-10-20T10:38:44.877334+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:38:44.877334+0000","last_undegraded":"2023-10-20T10:38:44.877334+0000","last_fullsized":"2023-10-20T10:38:44.877334+0000","mapping_epoch":16,"log_start":"137'543326","ondisk_log_start":"137'543326","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":4398,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 600s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7254863891,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":547249,"num_read_kb":2156960,"num_write":546535,"num_write_kb":2186134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":223,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-20T10:37:29.466999+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:37:29.466999+0000","last_peered":"2023-10-20T10:37:29.466999+0000","last_clean":"2023-10-20T10:37:29.466999+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:37:29.466999+0000","last_undegraded":"2023-10-20T10:37:29.466999+0000","last_fullsized":"2023-10-20T10:37:29.466999+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":317,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:37:53.361632+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:37:53.361632+0000","last_peered":"2023-10-20T10:37:53.361632+0000","last_clean":"2023-10-20T10:37:53.361632+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:37:53.361632+0000","last_undegraded":"2023-10-20T10:37:53.361632+0000","last_fullsized":"2023-10-20T10:37:53.361632+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28810432531,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":2172068,"num_read_kb":8562060,"num_write":2167644,"num_write_kb":8670570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17755574272,"data_stored":31138328614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40098,"ondisk_log_size":40098,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53442,"internal_metadata":0},"log_size":30839,"ondisk_log_size":30839,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739392,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10424836,"kb_used_data":8670204,"kb_used_omap":1970,"kb_used_meta":1752653,"kb_avail":83947004,"statfs":{"total":96636764160,"available":85961732096,"internally_reserved":0,"allocated":8878288896,"data_stored":15568753175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2017535,"internal_metadata":1794717441},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:38:32 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.373,"15min":0.441},"min":{"1min":0.17399999999999999,"5min":0.17100000000000001,"15min":0.16300000000000001},"max":{"1min":0.63400000000000001,"5min":4.8129999999999997,"15min":6.2430000000000003},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.46999999999999997,"15min":0.496},"min":{"1min":0.20399999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":3.9380000000000002,"5min":4.8540000000000001,"15min":5.048},"last":0.33600000000000002}]},{"osd":1,"last update":"Fri Oct 20 10:38:32 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.45200000000000001,"15min":0.49099999999999999},"min":{"1min":0.217,"5min":0.191,"15min":0.186},"max":{"1min":3.9689999999999999,"5min":4.476,"15min":6.1509999999999998},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.65100000000000002,"15min":0.55400000000000005},"min":{"1min":0.25800000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.1059999999999999,"5min":12.16,"15min":12.16},"last":0.26800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739392,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6027456,"kb_used_data":4253112,"kb_used_omap":902,"kb_used_meta":1773433,"kb_avail":88344384,"statfs":{"total":96636764160,"available":90464649216,"internally_reserved":0,"allocated":4355186688,"data_stored":7619313035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":924229,"internal_metadata":1815995835},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,8,9,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:38:28 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.41399999999999998,"15min":0.432},"min":{"1min":0.19500000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":1.3620000000000001,"5min":5.0750000000000002,"15min":5.0750000000000002},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.48099999999999998,"15min":0.45800000000000002},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.185},"max":{"1min":12.724,"5min":12.724,"15min":12.724},"last":0.20699999999999999}]},{"osd":2,"last update":"Fri Oct 20 10:38:28 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.39200000000000002,"15min":0.47799999999999998},"min":{"1min":0.22700000000000001,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":0.54400000000000004,"5min":4.5899999999999999,"15min":7.6459999999999999},"last":0.31},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.44500000000000001,"15min":0.52600000000000002},"min":{"1min":0.28000000000000003,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.57099999999999995,"5min":4.625,"15min":7.6820000000000004},"last":0.372}]}]},{"osd":0,"up_from":8,"seq":34359739394,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6153212,"kb_used_data":4421364,"kb_used_omap":1025,"kb_used_meta":1730814,"kb_avail":88218628,"statfs":{"total":96636764160,"available":90335875072,"internally_reserved":0,"allocated":4527476736,"data_stored":7952582396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1049977,"internal_metadata":1772354183},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:38:19 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.45700000000000002,"15min":0.58199999999999996},"min":{"1min":0.17100000000000001,"5min":0.158,"15min":0.158},"max":{"1min":6.2939999999999996,"5min":6.2939999999999996,"15min":8.0869999999999997},"last":2.1080000000000001},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.36799999999999999,"15min":0.45500000000000002},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.188},"max":{"1min":1.1080000000000001,"5min":6.1449999999999996,"15min":6.875},"last":0.20799999999999999}]},{"osd":2,"last update":"Fri Oct 20 10:38:19 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.61499999999999999,"15min":0.54000000000000004},"min":{"1min":0.25900000000000001,"5min":0.223,"15min":0.20200000000000001},"max":{"1min":6.2389999999999999,"5min":14.673,"15min":14.673},"last":0.314},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.65800000000000003,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.193,"15min":0.193},"max":{"1min":6.1779999999999999,"5min":15.686,"15min":15.686},"last":0.29199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13921,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4525703168,"data_stored":7951806464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4352970752,"data_stored":7618244627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8876900352,"data_stored":15568277523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:38:45.879 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:38:45.879 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:38:59.973 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:38:59.971+0000 7f3810eae640 -1 reset not still connected to 0x5632c20b7ba0 2023-10-20T10:38:59.973 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:38:59.971+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0000 2023-10-20T10:38:59.974 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:38:59.971+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:38:59.974 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:38:59.971+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab4f70 2023-10-20T10:38:59.974 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:38:59.971+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5450 2023-10-20T10:38:59.975 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:38:59.971+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:38:59.975 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:38:59.971+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5ad0 2023-10-20T10:38:59.975 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:38:59.971+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2000 2023-10-20T10:38:59.976 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:38:59.971+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2340 2023-10-20T10:38:59.976 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:38:59.971+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2410 2023-10-20T10:38:59.976 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:38:59.971+0000 7f3810eae640 -1 reset not still connected to 0x5632c30e2750 2023-10-20T10:39:05.881 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:39:06.511 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:39:06.513 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:39:06.540 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2704,"stamp":"2023-10-20T10:39:05.442958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28843495939,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2218704,"num_read_kb":8626561,"num_write":2176544,"num_write_kb":8704427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71204,"ondisk_log_size":71204,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22865876,"kb_used_data":17405948,"kb_used_omap":3898,"kb_used_meta":5456005,"kb_avail":260249644,"statfs":{"total":289910292480,"available":266495635456,"internally_reserved":0,"allocated":17823690752,"data_stored":31203374609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3991741,"internal_metadata":5586949955},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,9,5,10,12,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16539648,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3875,"num_read_kb":15448,"num_write":4030,"num_write_kb":16120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001621"},"pg_stats":[{"pgid":"2.7","version":"139'11089","reported_seq":16292,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:39:02.798009+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:39:02.798009+0000","last_peered":"2023-10-20T10:39:02.798009+0000","last_clean":"2023-10-20T10:39:02.798009+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:39:02.798009+0000","last_undegraded":"2023-10-20T10:39:02.798009+0000","last_fullsized":"2023-10-20T10:39:02.798009+0000","mapping_epoch":13,"log_start":"50'1000","ondisk_log_start":"50'1000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"139'21823","reported_seq":33196,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:39:02.799428+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:39:02.799428+0000","last_peered":"2023-10-20T10:39:02.799428+0000","last_clean":"2023-10-20T10:39:02.799428+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:39:02.799428+0000","last_undegraded":"2023-10-20T10:39:02.799428+0000","last_fullsized":"2023-10-20T10:39:02.799428+0000","mapping_epoch":13,"log_start":"118'11800","ondisk_log_start":"118'11800","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":225,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:37:53.361726+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:37:53.361726+0000","last_peered":"2023-10-20T10:37:53.361726+0000","last_clean":"2023-10-20T10:37:53.361726+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:37:53.361726+0000","last_undegraded":"2023-10-20T10:37:53.361726+0000","last_fullsized":"2023-10-20T10:37:53.361726+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"139'11047","reported_seq":17025,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:39:02.798549+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:39:02.798549+0000","last_peered":"2023-10-20T10:39:02.798549+0000","last_clean":"2023-10-20T10:39:02.798549+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:39:02.798549+0000","last_undegraded":"2023-10-20T10:39:02.798549+0000","last_fullsized":"2023-10-20T10:39:02.798549+0000","mapping_epoch":13,"log_start":"50'1000","ondisk_log_start":"50'1000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"139'565784","reported_seq":1122555,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:39:04.286952+0000","last_change":"2023-10-20T10:31:19.489466+0000","last_active":"2023-10-20T10:39:04.286952+0000","last_peered":"2023-10-20T10:39:04.286952+0000","last_clean":"2023-10-20T10:39:04.286952+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:39:04.286952+0000","last_undegraded":"2023-10-20T10:39:04.286952+0000","last_fullsized":"2023-10-20T10:39:04.286952+0000","mapping_epoch":16,"log_start":"137'555732","ondisk_log_start":"137'555732","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'519713","last_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_deep_scrub":"132'519713","last_deep_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_clean_scrub_stamp":"2023-10-20T10:31:19.489275+0000","objects_scrubbed":4836,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":696,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:01:22.001899+0000","scrub_duration":695.57415461999994,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7410311168,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":559626,"num_read_kb":2207516,"num_write":558572,"num_write_kb":2234288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":319,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:37:53.359027+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:37:53.359027+0000","last_peered":"2023-10-20T10:37:53.359027+0000","last_clean":"2023-10-20T10:37:53.359027+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:37:53.359027+0000","last_undegraded":"2023-10-20T10:37:53.359027+0000","last_fullsized":"2023-10-20T10:37:53.359027+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"139'524841","reported_seq":1041809,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:39:04.985156+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:39:04.985156+0000","last_peered":"2023-10-20T10:39:04.985156+0000","last_clean":"2023-10-20T10:39:04.985156+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:39:04.985156+0000","last_undegraded":"2023-10-20T10:39:04.985156+0000","last_fullsized":"2023-10-20T10:39:04.985156+0000","mapping_epoch":16,"log_start":"137'514742","ondisk_log_start":"137'514742","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6870913024,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":519656,"num_read_kb":2047688,"num_write":518097,"num_write_kb":2072388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"135'338","reported_seq":7122,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:39:02.799425+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:39:02.799425+0000","last_peered":"2023-10-20T10:39:02.799425+0000","last_clean":"2023-10-20T10:39:02.799425+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:39:02.799425+0000","last_undegraded":"2023-10-20T10:39:02.799425+0000","last_fullsized":"2023-10-20T10:39:02.799425+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"139'557581","reported_seq":1106209,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:39:04.227919+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:39:04.227919+0000","last_peered":"2023-10-20T10:39:04.227919+0000","last_clean":"2023-10-20T10:39:04.227919+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:39:04.227919+0000","last_undegraded":"2023-10-20T10:39:04.227919+0000","last_fullsized":"2023-10-20T10:39:04.227919+0000","mapping_epoch":16,"log_start":"137'547536","ondisk_log_start":"137'547536","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7298715648,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":551522,"num_read_kb":2173764,"num_write":550343,"num_write_kb":2201372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"135'315","reported_seq":6970,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:39:02.798469+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:39:02.798469+0000","last_peered":"2023-10-20T10:39:02.798469+0000","last_clean":"2023-10-20T10:39:02.798469+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:39:02.798469+0000","last_undegraded":"2023-10-20T10:39:02.798469+0000","last_fullsized":"2023-10-20T10:39:02.798469+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"139'555428","reported_seq":1101982,"reported_epoch":139,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:39:04.981673+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:39:04.981673+0000","last_peered":"2023-10-20T10:39:04.981673+0000","last_clean":"2023-10-20T10:39:04.981673+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:39:04.981673+0000","last_undegraded":"2023-10-20T10:39:04.981673+0000","last_fullsized":"2023-10-20T10:39:04.981673+0000","mapping_epoch":16,"log_start":"137'545426","ondisk_log_start":"137'545426","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":4543,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 620s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7262961683,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":549229,"num_read_kb":2164872,"num_write":548518,"num_write_kb":2194066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":224,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:37:53.361630+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:37:53.361630+0000","last_peered":"2023-10-20T10:37:53.361630+0000","last_clean":"2023-10-20T10:37:53.361630+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:37:53.361630+0000","last_undegraded":"2023-10-20T10:37:53.361630+0000","last_fullsized":"2023-10-20T10:37:53.361630+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":318,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:38:04.685002+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:38:04.685002+0000","last_peered":"2023-10-20T10:38:04.685002+0000","last_clean":"2023-10-20T10:38:04.685002+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:38:04.685002+0000","last_undegraded":"2023-10-20T10:38:04.685002+0000","last_fullsized":"2023-10-20T10:38:04.685002+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28842901523,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":2180033,"num_read_kb":8593840,"num_write":2175530,"num_write_kb":8702114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17819955200,"data_stored":31202709542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40198,"ondisk_log_size":40198,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53442,"internal_metadata":0},"log_size":30907,"ondisk_log_size":30907,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739396,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10540712,"kb_used_data":8700000,"kb_used_omap":1970,"kb_used_meta":1838733,"kb_avail":83831128,"statfs":{"total":96636764160,"available":85843075072,"internally_reserved":0,"allocated":8908800000,"data_stored":15599260136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2017535,"internal_metadata":1882863361},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:38:32 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.373,"15min":0.441},"min":{"1min":0.17399999999999999,"5min":0.17100000000000001,"15min":0.16300000000000001},"max":{"1min":0.63400000000000001,"5min":4.8129999999999997,"15min":6.2430000000000003},"last":0.316},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.46999999999999997,"15min":0.496},"min":{"1min":0.20399999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":3.9380000000000002,"5min":4.8540000000000001,"15min":5.048},"last":0.30199999999999999}]},{"osd":1,"last update":"Fri Oct 20 10:38:32 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.45200000000000001,"15min":0.49099999999999999},"min":{"1min":0.217,"5min":0.191,"15min":0.186},"max":{"1min":3.9689999999999999,"5min":4.476,"15min":6.1509999999999998},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.65100000000000002,"15min":0.55400000000000005},"min":{"1min":0.25800000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.1059999999999999,"5min":12.16,"15min":12.16},"last":0.27400000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739396,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6097668,"kb_used_data":4268604,"kb_used_omap":902,"kb_used_meta":1828153,"kb_avail":88274172,"statfs":{"total":96636764160,"available":90392752128,"internally_reserved":0,"allocated":4371050496,"data_stored":7635172700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":924229,"internal_metadata":1872029115},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,1,5,8,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:38:28 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.41399999999999998,"15min":0.432},"min":{"1min":0.19500000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":1.3620000000000001,"5min":5.0750000000000002,"15min":5.0750000000000002},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.48099999999999998,"15min":0.45800000000000002},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.185},"max":{"1min":12.724,"5min":12.724,"15min":12.724},"last":0.247}]},{"osd":2,"last update":"Fri Oct 20 10:38:28 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.39200000000000002,"15min":0.47799999999999998},"min":{"1min":0.22700000000000001,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":0.54400000000000004,"5min":4.5899999999999999,"15min":7.6459999999999999},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.44500000000000001,"15min":0.52600000000000002},"min":{"1min":0.28000000000000003,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.57099999999999995,"5min":4.625,"15min":7.6820000000000004},"last":0.33700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739398,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6227496,"kb_used_data":4437344,"kb_used_omap":1025,"kb_used_meta":1789118,"kb_avail":88144344,"statfs":{"total":96636764160,"available":90259808256,"internally_reserved":0,"allocated":4543840256,"data_stored":7968941773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1049977,"internal_metadata":1832057479},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:38:19 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.45700000000000002,"15min":0.58199999999999996},"min":{"1min":0.17100000000000001,"5min":0.158,"15min":0.158},"max":{"1min":6.2939999999999996,"5min":6.2939999999999996,"15min":8.0869999999999997},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.36799999999999999,"15min":0.45500000000000002},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.188},"max":{"1min":1.1080000000000001,"5min":6.1449999999999996,"15min":6.875},"last":0.26900000000000002}]},{"osd":2,"last update":"Fri Oct 20 10:38:19 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.61499999999999999,"15min":0.54000000000000004},"min":{"1min":0.25900000000000001,"5min":0.223,"15min":0.20200000000000001},"max":{"1min":6.2389999999999999,"5min":14.673,"15min":14.673},"last":0.307},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.65800000000000003,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.193,"15min":0.193},"max":{"1min":6.1779999999999999,"5min":15.686,"15min":15.686},"last":0.249}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13921,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4542132224,"data_stored":7968235520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4368850944,"data_stored":7634124819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8908972032,"data_stored":15600349203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:39:06.542 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:39:06.543 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:39:18.909 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:39:18.895+0000 7f3810eae640 -1 reset not still connected to 0x5632c29c0ea0 2023-10-20T10:39:18.917 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:39:18.915+0000 7f3810eae640 -1 reset not still connected to 0x5632c2ab5860 2023-10-20T10:39:26.544 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:39:27.345 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:39:27.346 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:39:27.389 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2714,"stamp":"2023-10-20T10:39:25.446395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28875530755,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2226662,"num_read_kb":8658285,"num_write":2184320,"num_write_kb":8735531,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71165,"ondisk_log_size":71165,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22687404,"kb_used_data":17468880,"kb_used_omap":3898,"kb_used_meta":5214597,"kb_avail":260428116,"statfs":{"total":289910292480,"available":266678390784,"internally_reserved":0,"allocated":17888133120,"data_stored":31267816977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3991741,"internal_metadata":5339748163},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,6,17,4,17,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24391680,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6025,"num_read_kb":24016,"num_write":5902,"num_write_kb":23608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002091"},"pg_stats":[{"pgid":"2.7","version":"139'11138","reported_seq":16353,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:39:24.748146+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:39:24.748146+0000","last_peered":"2023-10-20T10:39:24.748146+0000","last_clean":"2023-10-20T10:39:24.748146+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:39:24.748146+0000","last_undegraded":"2023-10-20T10:39:24.748146+0000","last_fullsized":"2023-10-20T10:39:24.748146+0000","mapping_epoch":13,"log_start":"50'1100","ondisk_log_start":"50'1100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"139'21920","reported_seq":33317,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:39:24.761963+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:39:24.761963+0000","last_peered":"2023-10-20T10:39:24.761963+0000","last_clean":"2023-10-20T10:39:24.761963+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:39:24.761963+0000","last_undegraded":"2023-10-20T10:39:24.761963+0000","last_fullsized":"2023-10-20T10:39:24.761963+0000","mapping_epoch":13,"log_start":"118'11900","ondisk_log_start":"118'11900","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":225,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:37:53.361726+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:37:53.361726+0000","last_peered":"2023-10-20T10:37:53.361726+0000","last_clean":"2023-10-20T10:37:53.361726+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:37:53.361726+0000","last_undegraded":"2023-10-20T10:37:53.361726+0000","last_fullsized":"2023-10-20T10:37:53.361726+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"139'11096","reported_seq":17086,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:39:24.748437+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:39:24.748437+0000","last_peered":"2023-10-20T10:39:24.748437+0000","last_clean":"2023-10-20T10:39:24.748437+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:39:24.748437+0000","last_undegraded":"2023-10-20T10:39:24.748437+0000","last_fullsized":"2023-10-20T10:39:24.748437+0000","mapping_epoch":13,"log_start":"50'1000","ondisk_log_start":"50'1000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"139'567816","reported_seq":1126621,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:39:24.332087+0000","last_change":"2023-10-20T10:31:19.489466+0000","last_active":"2023-10-20T10:39:24.332087+0000","last_peered":"2023-10-20T10:39:24.332087+0000","last_clean":"2023-10-20T10:39:24.332087+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:39:24.332087+0000","last_undegraded":"2023-10-20T10:39:24.332087+0000","last_fullsized":"2023-10-20T10:39:24.332087+0000","mapping_epoch":16,"log_start":"137'557732","ondisk_log_start":"137'557732","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'519713","last_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_deep_scrub":"132'519713","last_deep_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_clean_scrub_stamp":"2023-10-20T10:31:19.489275+0000","objects_scrubbed":4836,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":696,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:01:22.001899+0000","scrub_duration":695.57415461999994,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7418638336,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":561661,"num_read_kb":2215620,"num_write":560604,"num_write_kb":2242416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":319,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:37:53.359027+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:37:53.359027+0000","last_peered":"2023-10-20T10:37:53.359027+0000","last_clean":"2023-10-20T10:37:53.359027+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:37:53.359027+0000","last_undegraded":"2023-10-20T10:37:53.359027+0000","last_fullsized":"2023-10-20T10:37:53.359027+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"139'526759","reported_seq":1045738,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:39:25.111740+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:39:25.111740+0000","last_peered":"2023-10-20T10:39:25.111740+0000","last_clean":"2023-10-20T10:39:25.111740+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:39:25.111740+0000","last_undegraded":"2023-10-20T10:39:25.111740+0000","last_fullsized":"2023-10-20T10:39:25.111740+0000","mapping_epoch":16,"log_start":"137'516742","ondisk_log_start":"137'516742","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6878830592,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":521666,"num_read_kb":2055708,"num_write":520015,"num_write_kb":2080060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"135'338","reported_seq":7139,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:39:23.278380+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:39:23.278380+0000","last_peered":"2023-10-20T10:39:23.278380+0000","last_clean":"2023-10-20T10:39:23.278380+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:39:23.278380+0000","last_undegraded":"2023-10-20T10:39:23.278380+0000","last_fullsized":"2023-10-20T10:39:23.278380+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"139'559494","reported_seq":1110095,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:39:24.326334+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:39:24.326334+0000","last_peered":"2023-10-20T10:39:24.326334+0000","last_clean":"2023-10-20T10:39:24.326334+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:39:24.326334+0000","last_undegraded":"2023-10-20T10:39:24.326334+0000","last_fullsized":"2023-10-20T10:39:24.326334+0000","mapping_epoch":16,"log_start":"137'549436","ondisk_log_start":"137'549436","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7306588160,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":553495,"num_read_kb":2181632,"num_write":552256,"num_write_kb":2209024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"135'315","reported_seq":6986,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:39:23.282036+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:39:23.282036+0000","last_peered":"2023-10-20T10:39:23.282036+0000","last_clean":"2023-10-20T10:39:23.282036+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:39:23.282036+0000","last_undegraded":"2023-10-20T10:39:23.282036+0000","last_fullsized":"2023-10-20T10:39:23.282036+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"139'557431","reported_seq":1105937,"reported_epoch":139,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-20T10:39:25.114517+0000","last_change":"2023-10-20T10:28:44.755190+0000","last_active":"2023-10-20T10:39:25.114517+0000","last_peered":"2023-10-20T10:39:25.114517+0000","last_clean":"2023-10-20T10:39:25.114517+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:39:25.114517+0000","last_undegraded":"2023-10-20T10:39:25.114517+0000","last_fullsized":"2023-10-20T10:39:25.114517+0000","mapping_epoch":16,"log_start":"137'547426","ondisk_log_start":"137'547426","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T09:13:43.751576+0000","last_clean_scrub_stamp":"2023-10-20T09:13:43.751576+0000","objects_scrubbed":4661,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 640s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7270879251,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":551169,"num_read_kb":2172604,"num_write":550431,"num_write_kb":2201718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":224,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:37:53.361630+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:37:53.361630+0000","last_peered":"2023-10-20T10:37:53.361630+0000","last_clean":"2023-10-20T10:37:53.361630+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:37:53.361630+0000","last_undegraded":"2023-10-20T10:37:53.361630+0000","last_fullsized":"2023-10-20T10:37:53.361630+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":318,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:38:04.685002+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:38:04.685002+0000","last_peered":"2023-10-20T10:38:04.685002+0000","last_clean":"2023-10-20T10:38:04.685002+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:38:04.685002+0000","last_undegraded":"2023-10-20T10:38:04.685002+0000","last_fullsized":"2023-10-20T10:38:04.685002+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28874936339,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":2187991,"num_read_kb":8625564,"num_write":2183306,"num_write_kb":8733218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17883541504,"data_stored":31266295846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40164,"ondisk_log_size":40164,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53442,"internal_metadata":0},"log_size":30902,"ondisk_log_size":30902,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739400,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10678968,"kb_used_data":8732016,"kb_used_omap":1970,"kb_used_meta":1944973,"kb_avail":83692872,"statfs":{"total":96636764160,"available":85701500928,"internally_reserved":0,"allocated":8941584384,"data_stored":15632044520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2017535,"internal_metadata":1991653121},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:38:32 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.373,"15min":0.441},"min":{"1min":0.17399999999999999,"5min":0.17100000000000001,"15min":0.16300000000000001},"max":{"1min":0.63400000000000001,"5min":4.8129999999999997,"15min":6.2430000000000003},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.46999999999999997,"15min":0.496},"min":{"1min":0.20399999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":3.9380000000000002,"5min":4.8540000000000001,"15min":5.048},"last":0.46400000000000002}]},{"osd":1,"last update":"Fri Oct 20 10:38:32 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.45200000000000001,"15min":0.49099999999999999},"min":{"1min":0.217,"5min":0.191,"15min":0.186},"max":{"1min":3.9689999999999999,"5min":4.476,"15min":6.1509999999999998},"last":0.5},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.65100000000000002,"15min":0.55400000000000005},"min":{"1min":0.25800000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.1059999999999999,"5min":12.16,"15min":12.16},"last":0.65700000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739400,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5733200,"kb_used_data":4283720,"kb_used_omap":902,"kb_used_meta":1448569,"kb_avail":88638640,"statfs":{"total":96636764160,"available":90765967360,"internally_reserved":0,"allocated":4386529280,"data_stored":7650651484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":924229,"internal_metadata":1483335099},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,9,4,13,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:38:28 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.41399999999999998,"15min":0.432},"min":{"1min":0.19500000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":1.3620000000000001,"5min":5.0750000000000002,"15min":5.0750000000000002},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.48099999999999998,"15min":0.45800000000000002},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.185},"max":{"1min":12.724,"5min":12.724,"15min":12.724},"last":0.47999999999999998}]},{"osd":2,"last update":"Fri Oct 20 10:38:28 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.39200000000000002,"15min":0.47799999999999998},"min":{"1min":0.22700000000000001,"5min":0.19800000000000001,"15min":0.19400000000000001},"max":{"1min":0.54400000000000004,"5min":4.5899999999999999,"15min":7.6459999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.44500000000000001,"15min":0.52600000000000002},"min":{"1min":0.28000000000000003,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.57099999999999995,"5min":4.625,"15min":7.6820000000000004},"last":0.42699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739402,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6275236,"kb_used_data":4453144,"kb_used_omap":1025,"kb_used_meta":1821054,"kb_avail":88096604,"statfs":{"total":96636764160,"available":90210922496,"internally_reserved":0,"allocated":4560019456,"data_stored":7985120973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1049977,"internal_metadata":1864759943},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:39:21 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.46300000000000002,"15min":0.5},"min":{"1min":0.17599999999999999,"5min":0.158,"15min":0.158},"max":{"1min":2.1080000000000001,"5min":6.2939999999999996,"15min":6.5449999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.309,"5min":0.36699999999999999,"15min":0.40600000000000003},"min":{"1min":0.20799999999999999,"5min":0.188,"15min":0.188},"max":{"1min":0.59199999999999997,"5min":6.1449999999999996,"15min":6.2229999999999999},"last":0.313}]},{"osd":2,"last update":"Fri Oct 20 10:39:21 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.621,"15min":0.54300000000000004},"min":{"1min":0.26100000000000001,"5min":0.22900000000000001,"15min":0.216},"max":{"1min":0.94099999999999995,"5min":14.673,"15min":14.673},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.63,"15min":0.51300000000000001},"min":{"1min":0.20000000000000001,"5min":0.193,"15min":0.193},"max":{"1min":1.966,"5min":15.686,"15min":15.686},"last":0.33600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13921,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4558274560,"data_stored":7984377856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4384542720,"data_stored":7649816595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8940724224,"data_stored":15632101395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:39:27.391 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-20T09:13:43.751576+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=9, tm_min=13, tm_sec=43, tm_wday=4, tm_yday=293, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=20, tm_hour=10, tm_min=19, tm_sec=38, tm_wday=4, tm_yday=293, tm_isdst=0) 2023-10-20T10:39:27.392 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-20T10:39:47.393 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T10:39:47.941 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-20T10:39:47.942 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2023-10-20T10:39:48.010 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":2725,"stamp":"2023-10-20T10:39:47.450343+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28908478979,"num_objects":19064,"num_object_clones":13824,"num_object_copies":38128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19064,"num_whiteouts":0,"num_read":2234535,"num_read_kb":8689677,"num_write":2192287,"num_write_kb":8767399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71262,"ondisk_log_size":71262,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22703484,"kb_used_data":17532324,"kb_used_omap":3988,"kb_used_meta":5167147,"kb_avail":260412036,"statfs":{"total":289910292480,"available":266661924864,"internally_reserved":0,"allocated":17953099776,"data_stored":31332783633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4084170,"internal_metadata":5291159094},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,4,9,36,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17342464,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4078,"num_read_kb":16256,"num_write":4186,"num_write_kb":16744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002129"},"pg_stats":[{"pgid":"2.7","version":"139'11201","reported_seq":16428,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:39:44.263925+0000","last_change":"2023-10-20T10:19:54.210547+0000","last_active":"2023-10-20T10:39:44.263925+0000","last_peered":"2023-10-20T10:39:44.263925+0000","last_clean":"2023-10-20T10:39:44.263925+0000","last_became_active":"2023-10-20T09:13:38.742506+0000","last_became_peered":"2023-10-20T09:13:38.742506+0000","last_unstale":"2023-10-20T10:39:44.263925+0000","last_undegraded":"2023-10-20T10:39:44.263925+0000","last_fullsized":"2023-10-20T10:39:44.263925+0000","mapping_epoch":13,"log_start":"52'1200","ondisk_log_start":"52'1200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8959","last_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_deep_scrub":"122'8959","last_deep_scrub_stamp":"2023-10-20T10:19:54.210286+0000","last_clean_scrub_stamp":"2023-10-20T10:19:54.210286+0000","objects_scrubbed":15,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T14:25:44.900853+0000","scrub_duration":0.11267978400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5557,"num_read_kb":4587,"num_write":138,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3290,"num_omap_keys":41,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"139'22046","reported_seq":33467,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:39:44.291228+0000","last_change":"2023-10-20T10:19:46.382462+0000","last_active":"2023-10-20T10:39:44.291228+0000","last_peered":"2023-10-20T10:39:44.291228+0000","last_clean":"2023-10-20T10:39:44.291228+0000","last_became_active":"2023-10-20T09:13:38.741374+0000","last_became_peered":"2023-10-20T09:13:38.741374+0000","last_unstale":"2023-10-20T10:39:44.291228+0000","last_undegraded":"2023-10-20T10:39:44.291228+0000","last_fullsized":"2023-10-20T10:39:44.291228+0000","mapping_epoch":13,"log_start":"118'12000","ondisk_log_start":"118'12000","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'17579","last_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_deep_scrub":"122'17579","last_deep_scrub_stamp":"2023-10-20T10:19:46.382238+0000","last_clean_scrub_stamp":"2023-10-20T10:19:46.382238+0000","objects_scrubbed":19,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:02:11.281000+0000","scrub_duration":0.17539958999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12667,"num_read_kb":11080,"num_write":231,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5386,"num_omap_keys":113,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"122'20","reported_seq":226,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:38:58.372197+0000","last_change":"2023-10-20T10:19:55.096749+0000","last_active":"2023-10-20T10:38:58.372197+0000","last_peered":"2023-10-20T10:38:58.372197+0000","last_clean":"2023-10-20T10:38:58.372197+0000","last_became_active":"2023-10-20T09:13:38.742090+0000","last_became_peered":"2023-10-20T09:13:38.742090+0000","last_unstale":"2023-10-20T10:38:58.372197+0000","last_undegraded":"2023-10-20T10:38:58.372197+0000","last_fullsized":"2023-10-20T10:38:58.372197+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'20","last_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_deep_scrub":"122'20","last_deep_scrub_stamp":"2023-10-20T10:19:55.096539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:55.096539+0000","objects_scrubbed":10,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T10:39:01.561494+0000","scrub_duration":0.012628297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"139'11159","reported_seq":17162,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:39:44.274415+0000","last_change":"2023-10-20T10:19:58.229193+0000","last_active":"2023-10-20T10:39:44.274415+0000","last_peered":"2023-10-20T10:39:44.274415+0000","last_clean":"2023-10-20T10:39:44.274415+0000","last_became_active":"2023-10-20T09:13:38.741729+0000","last_became_peered":"2023-10-20T09:13:38.741729+0000","last_unstale":"2023-10-20T10:39:44.274415+0000","last_undegraded":"2023-10-20T10:39:44.274415+0000","last_fullsized":"2023-10-20T10:39:44.274415+0000","mapping_epoch":13,"log_start":"50'1100","ondisk_log_start":"50'1100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8931","last_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_deep_scrub":"122'8931","last_deep_scrub_stamp":"2023-10-20T10:19:58.229007+0000","last_clean_scrub_stamp":"2023-10-20T10:19:58.229007+0000","objects_scrubbed":13,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T12:06:56.267343+0000","scrub_duration":0.104725491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":6198,"num_read_kb":5238,"num_write":140,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3168,"num_omap_keys":78,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"139'569832","reported_seq":1130704,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:39:44.390728+0000","last_change":"2023-10-20T10:31:19.489466+0000","last_active":"2023-10-20T10:39:44.390728+0000","last_peered":"2023-10-20T10:39:44.390728+0000","last_clean":"2023-10-20T10:39:44.390728+0000","last_became_active":"2023-10-20T09:13:44.810217+0000","last_became_peered":"2023-10-20T09:13:44.810217+0000","last_unstale":"2023-10-20T10:39:44.390728+0000","last_undegraded":"2023-10-20T10:39:44.390728+0000","last_fullsized":"2023-10-20T10:39:44.390728+0000","mapping_epoch":16,"log_start":"138'559732","ondisk_log_start":"138'559732","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'519713","last_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_deep_scrub":"132'519713","last_deep_scrub_stamp":"2023-10-20T10:31:19.489275+0000","last_clean_scrub_stamp":"2023-10-20T10:31:19.489275+0000","objects_scrubbed":4836,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":696,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T17:01:22.001899+0000","scrub_duration":695.57415461999994,"objects_trimmed":0,"snaptrim_duration":0.11489993599999999,"stat_sum":{"num_bytes":7426904064,"num_objects":4836,"num_object_clones":3520,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":563725,"num_read_kb":2223856,"num_write":562620,"num_write_kb":2250480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"122'49","reported_seq":320,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:38:58.358612+0000","last_change":"2023-10-20T10:19:41.970776+0000","last_active":"2023-10-20T10:38:58.358612+0000","last_peered":"2023-10-20T10:38:58.358612+0000","last_clean":"2023-10-20T10:38:58.358612+0000","last_became_active":"2023-10-20T09:13:38.741130+0000","last_became_peered":"2023-10-20T09:13:38.741130+0000","last_unstale":"2023-10-20T10:38:58.358612+0000","last_undegraded":"2023-10-20T10:38:58.358612+0000","last_fullsized":"2023-10-20T10:38:58.358612+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'49","last_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_deep_scrub":"122'49","last_deep_scrub_stamp":"2023-10-20T10:19:41.970539+0000","last_clean_scrub_stamp":"2023-10-20T10:19:41.970539+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T13:48:43.467576+0000","scrub_duration":0.062478038999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":601,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":105,"num_read_kb":111,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"139'528719","reported_seq":1049556,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:39:45.186275+0000","last_change":"2023-10-20T10:28:42.030801+0000","last_active":"2023-10-20T10:39:45.186275+0000","last_peered":"2023-10-20T10:39:45.186275+0000","last_clean":"2023-10-20T10:39:45.186275+0000","last_became_active":"2023-10-20T09:13:44.810389+0000","last_became_peered":"2023-10-20T09:13:44.810389+0000","last_unstale":"2023-10-20T10:39:45.186275+0000","last_undegraded":"2023-10-20T10:39:45.186275+0000","last_fullsized":"2023-10-20T10:39:45.186275+0000","mapping_epoch":16,"log_start":"138'518642","ondisk_log_start":"138'518642","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'468866","last_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_deep_scrub":"130'468866","last_deep_scrub_stamp":"2023-10-20T10:28:42.030647+0000","last_clean_scrub_stamp":"2023-10-20T10:28:42.030647+0000","objects_scrubbed":4512,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":523,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:07:18.988481+0000","scrub_duration":522.87640117499996,"objects_trimmed":0,"snaptrim_duration":0.011527630000000001,"stat_sum":{"num_bytes":6886981632,"num_objects":4512,"num_object_clones":3293,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":523527,"num_read_kb":2063128,"num_write":521975,"num_write_kb":2087900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"135'338","reported_seq":7151,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:39:38.644173+0000","last_change":"2023-10-20T10:19:44.071273+0000","last_active":"2023-10-20T10:39:38.644173+0000","last_peered":"2023-10-20T10:39:38.644173+0000","last_clean":"2023-10-20T10:39:38.644173+0000","last_became_active":"2023-10-20T09:13:38.740211+0000","last_became_peered":"2023-10-20T09:13:38.740211+0000","last_unstale":"2023-10-20T10:39:38.644173+0000","last_undegraded":"2023-10-20T10:39:38.644173+0000","last_fullsized":"2023-10-20T10:39:38.644173+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'326","last_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_deep_scrub":"122'326","last_deep_scrub_stamp":"2023-10-20T10:19:44.071028+0000","last_clean_scrub_stamp":"2023-10-20T10:19:44.071028+0000","objects_scrubbed":17,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T16:26:27.836943+0000","scrub_duration":0.18061823199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":621,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7363,"num_read_kb":6085,"num_write":166,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4458,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"139'561469","reported_seq":1114018,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:39:44.407547+0000","last_change":"2023-10-20T10:30:19.969824+0000","last_active":"2023-10-20T10:39:44.407547+0000","last_peered":"2023-10-20T10:39:44.407547+0000","last_clean":"2023-10-20T10:39:44.407547+0000","last_became_active":"2023-10-20T09:13:44.810555+0000","last_became_peered":"2023-10-20T09:13:44.810555+0000","last_unstale":"2023-10-20T10:39:44.407547+0000","last_undegraded":"2023-10-20T10:39:44.407547+0000","last_fullsized":"2023-10-20T10:39:44.407547+0000","mapping_epoch":16,"log_start":"138'551436","ondisk_log_start":"138'551436","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'506858","last_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_deep_scrub":"131'506858","last_deep_scrub_stamp":"2023-10-20T10:30:19.969635+0000","last_clean_scrub_stamp":"2023-10-20T10:30:19.969635+0000","objects_scrubbed":4833,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":636,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:31:25.125756+0000","scrub_duration":635.01865714500002,"objects_trimmed":0,"snaptrim_duration":0.027113927999999999,"stat_sum":{"num_bytes":7314870272,"num_objects":4833,"num_object_clones":3537,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":555443,"num_read_kb":2189400,"num_write":554231,"num_write_kb":2216924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"135'315","reported_seq":6998,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:39:38.638550+0000","last_change":"2023-10-20T10:19:45.961719+0000","last_active":"2023-10-20T10:39:38.638550+0000","last_peered":"2023-10-20T10:39:38.638550+0000","last_clean":"2023-10-20T10:39:38.638550+0000","last_became_active":"2023-10-20T09:13:38.738467+0000","last_became_peered":"2023-10-20T09:13:38.738467+0000","last_unstale":"2023-10-20T10:39:38.638550+0000","last_undegraded":"2023-10-20T10:39:38.638550+0000","last_fullsized":"2023-10-20T10:39:38.638550+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'303","last_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_deep_scrub":"122'303","last_deep_scrub_stamp":"2023-10-20T10:19:45.961480+0000","last_clean_scrub_stamp":"2023-10-20T10:19:45.961480+0000","objects_scrubbed":15,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T21:33:27.633117+0000","scrub_duration":0.029968907999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6610,"num_read_kb":5509,"num_write":145,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3373,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"139'559525","reported_seq":1110044,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:39:45.167069+0000","last_change":"2023-10-20T10:39:36.642099+0000","last_active":"2023-10-20T10:39:45.167069+0000","last_peered":"2023-10-20T10:39:45.167069+0000","last_clean":"2023-10-20T10:39:45.167069+0000","last_became_active":"2023-10-20T09:13:44.810371+0000","last_became_peered":"2023-10-20T09:13:44.810371+0000","last_unstale":"2023-10-20T10:39:45.167069+0000","last_undegraded":"2023-10-20T10:39:45.167069+0000","last_fullsized":"2023-10-20T10:39:45.167069+0000","mapping_epoch":16,"log_start":"138'549426","ondisk_log_start":"138'549426","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'558566","last_scrub_stamp":"2023-10-20T10:39:36.641943+0000","last_deep_scrub":"139'558566","last_deep_scrub_stamp":"2023-10-20T10:39:36.641943+0000","last_clean_scrub_stamp":"2023-10-20T10:39:36.641943+0000","objects_scrubbed":4764,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":652,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T18:29:27.934207+0000","scrub_duration":651.85992057099998,"objects_trimmed":0,"snaptrim_duration":0.17777606700000001,"stat_sum":{"num_bytes":7279128595,"num_objects":4764,"num_object_clones":3474,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4764,"num_whiteouts":0,"num_read":553169,"num_read_kb":2180572,"num_write":552447,"num_write_kb":2209782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"122'26","reported_seq":225,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-20T10:38:58.372152+0000","last_change":"2023-10-20T10:19:51.179435+0000","last_active":"2023-10-20T10:38:58.372152+0000","last_peered":"2023-10-20T10:38:58.372152+0000","last_clean":"2023-10-20T10:38:58.372152+0000","last_became_active":"2023-10-20T09:13:38.738462+0000","last_became_peered":"2023-10-20T09:13:38.738462+0000","last_unstale":"2023-10-20T10:38:58.372152+0000","last_undegraded":"2023-10-20T10:38:58.372152+0000","last_fullsized":"2023-10-20T10:38:58.372152+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'26","last_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_deep_scrub":"122'26","last_deep_scrub_stamp":"2023-10-20T10:19:51.178428+0000","last_clean_scrub_stamp":"2023-10-20T10:19:51.178428+0000","objects_scrubbed":13,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T22:14:45.626817+0000","scrub_duration":0.062332712999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"122'99","reported_seq":318,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-20T10:38:04.685002+0000","last_change":"2023-10-20T10:19:42.469749+0000","last_active":"2023-10-20T10:38:04.685002+0000","last_peered":"2023-10-20T10:38:04.685002+0000","last_clean":"2023-10-20T10:38:04.685002+0000","last_became_active":"2023-10-20T09:13:35.014023+0000","last_became_peered":"2023-10-20T09:13:35.014023+0000","last_unstale":"2023-10-20T10:38:04.685002+0000","last_undegraded":"2023-10-20T10:38:04.685002+0000","last_fullsized":"2023-10-20T10:38:04.685002+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'99","last_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_deep_scrub":"122'99","last_deep_scrub_stamp":"2023-10-20T10:19:42.469516+0000","last_clean_scrub_stamp":"2023-10-20T10:19:42.469516+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T11:57:56.799628+0000","scrub_duration":0.100684335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28907884563,"num_objects":18945,"num_object_clones":13824,"num_object_copies":37890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18945,"num_whiteouts":0,"num_read":2195864,"num_read_kb":8656956,"num_write":2191273,"num_write_kb":8765086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17948782592,"data_stored":31331536934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40309,"ondisk_log_size":40309,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4048,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":38565,"num_read_kb":32630,"num_write":877,"num_write_kb":419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":21035,"num_omap_keys":357,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":8096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53442,"internal_metadata":0},"log_size":30854,"ondisk_log_size":30854,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739404,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10544364,"kb_used_data":8763812,"kb_used_omap":2060,"kb_used_meta":1778483,"kb_avail":83827476,"statfs":{"total":96636764160,"available":85839335424,"internally_reserved":0,"allocated":8974143488,"data_stored":15664603624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2109964,"internal_metadata":1821167092},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,5,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:39:34 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.376,"15min":0.436},"min":{"1min":0.18099999999999999,"5min":0.17100000000000001,"15min":0.16300000000000001},"max":{"1min":0.96199999999999997,"5min":4.8129999999999997,"15min":6.2430000000000003},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.51400000000000001,"15min":0.51100000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.14799999999999999},"max":{"1min":2.5819999999999999,"5min":4.8540000000000001,"15min":5.048},"last":0.371}]},{"osd":1,"last update":"Fri Oct 20 10:39:34 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.48499999999999999,"15min":0.497},"min":{"1min":0.192,"5min":0.192,"15min":0.186},"max":{"1min":2.7269999999999999,"5min":4.476,"15min":6.1509999999999998},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.68799999999999994,"15min":0.56299999999999994},"min":{"1min":0.248,"5min":0.22700000000000001,"15min":0.17000000000000001},"max":{"1min":2.621,"5min":12.16,"15min":12.16},"last":0.33900000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739404,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5803112,"kb_used_data":4299488,"kb_used_omap":902,"kb_used_meta":1502713,"kb_avail":88568728,"statfs":{"total":96636764160,"available":90694377472,"internally_reserved":0,"allocated":4402675712,"data_stored":7666797916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":924229,"internal_metadata":1538778555},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,29,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 10:39:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.42799999999999999,"15min":0.434},"min":{"1min":0.22500000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.5999999999999996,"5min":4.5999999999999996,"15min":5.0750000000000002},"last":0.186},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.61399999999999999,"15min":0.48699999999999999},"min":{"1min":0.20399999999999999,"5min":0.188,"15min":0.185},"max":{"1min":6.7000000000000002,"5min":12.724,"15min":12.724},"last":0.28199999999999997}]},{"osd":2,"last update":"Fri Oct 20 10:39:28 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.46400000000000002},"min":{"1min":0.25,"5min":0.20399999999999999,"15min":0.19400000000000001},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":7.6459999999999999},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.40200000000000002,"15min":0.51200000000000001},"min":{"1min":0.309,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":7.6820000000000004},"last":0.26700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739406,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6356008,"kb_used_data":4469024,"kb_used_omap":1025,"kb_used_meta":1885950,"kb_avail":88015832,"statfs":{"total":96636764160,"available":90128211968,"internally_reserved":0,"allocated":4576280576,"data_stored":8001382093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1049977,"internal_metadata":1931213447},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,3,3,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 10:39:21 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.46300000000000002,"15min":0.5},"min":{"1min":0.17599999999999999,"5min":0.158,"15min":0.158},"max":{"1min":2.1080000000000001,"5min":6.2939999999999996,"15min":6.5449999999999999},"last":0.249},{"interface":"front","average":{"1min":0.309,"5min":0.36699999999999999,"15min":0.40600000000000003},"min":{"1min":0.20799999999999999,"5min":0.188,"15min":0.188},"max":{"1min":0.59199999999999997,"5min":6.1449999999999996,"15min":6.2229999999999999},"last":0.28299999999999997}]},{"osd":2,"last update":"Fri Oct 20 10:39:21 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.621,"15min":0.54300000000000004},"min":{"1min":0.26100000000000001,"5min":0.22900000000000001,"15min":0.216},"max":{"1min":0.94099999999999995,"5min":14.673,"15min":14.673},"last":0.312},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.63,"15min":0.51300000000000001},"min":{"1min":0.20000000000000001,"5min":0.193,"15min":0.193},"max":{"1min":1.966,"5min":15.686,"15min":15.686},"last":0.224}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16388,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":4048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23133,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13921,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4574633984,"data_stored":8000737280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4400828416,"data_stored":7666102291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8973320192,"data_stored":15664697363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T10:39:48.013 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-20T10:39:48.641 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-20T10:39:48.642 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-20T10:39:48.642 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-20T10:39:48.643 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T10:39:52.314 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-20T10:39:54.010 INFO:tasks.ceph.osd.1.smithi150.stderr:2023-10-20T10:39:54.006+0000 7f3810eae640 -1 reset not still connected to 0x5632e4466000 2023-10-20T10:39:54.745 INFO:tasks.ceph.osd.0:Stopped 2023-10-20T10:39:54.746 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-20T10:39:54.746 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T10:39:57.617 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-20T10:39:57.617 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-20T10:40:00.848 INFO:tasks.ceph.osd.1:Stopped 2023-10-20T10:40:00.849 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-20T10:40:00.849 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T10:40:02.819 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-20T10:40:02.820 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-10-20T10:40:06.951 INFO:tasks.ceph.osd.2:Stopped 2023-10-20T10:40:06.951 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-20T10:40:06.952 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-20T10:40:06.952 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T10:40:07.922 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2023-10-20T10:40:07.922 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-10-20T10:40:13.024 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~5s 2023-10-20T10:40:13.054 INFO:tasks.ceph.mgr.x:Stopped 2023-10-20T10:40:13.055 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-20T10:40:13.055 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-20T10:40:13.056 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T10:40:18.026 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-10-20T10:40:18.026 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2023-10-20T10:40:19.157 INFO:tasks.ceph.mon.a:Stopped 2023-10-20T10:40:19.158 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-20T10:40:19.159 DEBUG:teuthology.orchestra.run.smithi150:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-10-20T10:40:19.244 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi150.front.sepia.ceph.com 2023-10-20T10:40:19.245 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-20T10:40:20.533 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi150.front.sepia.ceph.com 2023-10-20T10:40:20.534 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-20T10:40:20.921 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi150.front.sepia.ceph.com 2023-10-20T10:40:20.921 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-20T10:40:21.557 INFO:tasks.ceph:Archiving mon data... 2023-10-20T10:40:21.559 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rraja-2023-10-20_08:52:26-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7432983/data/mon.a.tgz 2023-10-20T10:40:21.560 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2023-10-20T10:40:21.581 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.M05uAPI1bU 2023-10-20T10:40:21.582 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.M05uAPI1bU 2023-10-20T10:40:21.751 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.M05uAPI1bU 2023-10-20T10:40:21.850 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.M05uAPI1bU is 569KB 2023-10-20T10:40:21.925 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.M05uAPI1bU 2023-10-20T10:40:21.931 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-20T10:40:21.932 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-20T10:40:22.044 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-20T10:40:22.044 DEBUG:teuthology.orchestra.run.smithi150:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-20T10:40:22.073 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-20T10:40:22.077 INFO:tasks.ceph:Archiving crash dumps... 2023-10-20T10:40:22.079 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/crash to /home/teuthworker/archive/rraja-2023-10-20_08:52:26-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7432983/remote/smithi150/crash 2023-10-20T10:40:22.081 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-20T10:40:22.134 INFO:tasks.ceph:Compressing logs... 2023-10-20T10:40:22.135 DEBUG:teuthology.orchestra.run.smithi150:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-20T10:40:22.211 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41296.log 2023-10-20T10:40:22.213 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47723.log 2023-10-20T10:40:22.213 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55537.log 2023-10-20T10:40:22.213 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41296.log: /var/log/ceph/ceph-client.admin.47723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40530.log 2023-10-20T10:40:22.214 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55537.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55537.log.gz 2023-10-20T10:40:22.214 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43473.log 2023-10-20T10:40:22.412 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52171.log 2023-10-20T10:40:22.413 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40530.log: /var/log/ceph/ceph-client.admin.43473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50254.log 2023-10-20T10:40:22.413 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56553.log 2023-10-20T10:40:22.413 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55724.log 2023-10-20T10:40:22.413 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50254.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52171.log.gz 2023-10-20T10:40:22.414 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50254.log.gz 2023-10-20T10:40:22.414 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56553.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56553.log.gz 2023-10-20T10:40:22.414 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56527.log 2023-10-20T10:40:22.415 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44142.log 2023-10-20T10:40:22.415 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55724.log.gz 2023-10-20T10:40:22.415 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43141.log 2023-10-20T10:40:22.415 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56527.log.gz 2023-10-20T10:40:22.416 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53602.log 2023-10-20T10:40:22.416 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55433.log 2023-10-20T10:40:22.416 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40513.log 2023-10-20T10:40:22.417 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53602.log.gz 2023-10-20T10:40:22.417 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47754.log 2023-10-20T10:40:22.417 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55433.log.gz 2023-10-20T10:40:22.417 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55060.log 2023-10-20T10:40:22.418 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55459.log 2023-10-20T10:40:22.418 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46959.log 2023-10-20T10:40:22.418 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55060.log.gz 2023-10-20T10:40:22.419 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45865.log 2023-10-20T10:40:22.419 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55459.log.gz 2023-10-20T10:40:22.419 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53722.log 2023-10-20T10:40:22.419 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40675.log 2023-10-20T10:40:22.420 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39550.log 2023-10-20T10:40:22.420 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53722.log.gz 2023-10-20T10:40:22.420 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54343.log 2023-10-20T10:40:22.421 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.39550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43089.log 2023-10-20T10:40:22.421 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39550.log.gz 2023-10-20T10:40:22.421 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54343.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40530.log.gz 2023-10-20T10:40:22.422 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52839.log 2023-10-20T10:40:22.422 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40513.log.gz 2023-10-20T10:40:22.422 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40675.log.gz 0.0% 0.0% 2023-10-20T10:40:22.422 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54343.log.gz -- replaced with /var/log/ceph/ceph-client.admin.41296.log.gz 2023-10-20T10:40:22.423 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-20T10:40:22.423 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45865.log.gz 2023-10-20T10:40:22.423 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43473.log.gz 2023-10-20T10:40:22.424 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43141.log.gz 2023-10-20T10:40:22.424 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46959.log.gz 2023-10-20T10:40:22.424 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44142.log.gz 2023-10-20T10:40:22.424 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45171.log 2023-10-20T10:40:22.425 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43089.log.gz 2023-10-20T10:40:22.425 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47754.log.gz 2023-10-20T10:40:22.425 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50193.log 2023-10-20T10:40:22.426 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52839.log.gz 2023-10-20T10:40:22.426 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47723.log.gz 2023-10-20T10:40:22.426 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46630.log 2023-10-20T10:40:22.426 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45171.log.gz 2023-10-20T10:40:22.427 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41390.log 2023-10-20T10:40:22.427 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50193.log.gz 2023-10-20T10:40:22.427 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46281.log 2023-10-20T10:40:22.427 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46630.log.gz 2023-10-20T10:40:22.428 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40800.log 2023-10-20T10:40:22.428 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41390.log.gz 2023-10-20T10:40:22.428 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46281.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.46281.log.gz 2023-10-20T10:40:22.429 INFO:teuthology.orchestra.run.smithi150.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45917.log 2023-10-20T10:40:22.429 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51805.log 2023-10-20T10:40:22.429 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40800.log.gz 2023-10-20T10:40:22.429 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40112.log 2023-10-20T10:40:22.430 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45917.log.gz 2023-10-20T10:40:22.430 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46383.log 2023-10-20T10:40:22.430 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51805.log.gz 2023-10-20T10:40:22.431 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46848.log 2023-10-20T10:40:22.431 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40112.log.gz 2023-10-20T10:40:22.431 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46741.log 2023-10-20T10:40:22.431 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46383.log.gz 2023-10-20T10:40:22.432 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52046.log 2023-10-20T10:40:22.432 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46848.log.gz 2023-10-20T10:40:22.432 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55750.log 2023-10-20T10:40:22.433 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46741.log.gz 2023-10-20T10:40:22.433 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49510.log 2023-10-20T10:40:22.433 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52046.log.gz 2023-10-20T10:40:22.433 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55750.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55750.log.gz 2023-10-20T10:40:22.434 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42895.log 2023-10-20T10:40:22.434 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56124.log 2023-10-20T10:40:22.434 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49510.log.gz 2023-10-20T10:40:22.434 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41328.log 2023-10-20T10:40:22.435 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42895.log.gz 2023-10-20T10:40:22.435 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53232.log 2023-10-20T10:40:22.435 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56124.log.gz 2023-10-20T10:40:22.436 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41328.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41328.log.gz 2023-10-20T10:40:22.436 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40928.log 2023-10-20T10:40:22.436 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51622.log 2023-10-20T10:40:22.437 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53232.log.gz 2023-10-20T10:40:22.437 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53201.log 2023-10-20T10:40:22.437 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40928.log.gz 2023-10-20T10:40:22.437 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51622.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51622.log.gz 2023-10-20T10:40:22.438 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49341.log 2023-10-20T10:40:22.438 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41161.log 2023-10-20T10:40:22.438 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53201.log.gz 2023-10-20T10:40:22.438 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46435.log 2023-10-20T10:40:22.439 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49341.log.gz 2023-10-20T10:40:22.439 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44283.log 2023-10-20T10:40:22.439 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41161.log.gz 2023-10-20T10:40:22.440 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39601.log 2023-10-20T10:40:22.440 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46435.log.gz 2023-10-20T10:40:22.440 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51896.log 2023-10-20T10:40:22.440 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44283.log.gz 2023-10-20T10:40:22.441 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.39601.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.39601.log.gz -- 2023-10-20T10:40:22.441 INFO:teuthology.orchestra.run.smithi150.stderr: /var/log/ceph/ceph-client.admin.48685.log 2023-10-20T10:40:22.441 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47416.log 2023-10-20T10:40:22.442 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51896.log.gz 2023-10-20T10:40:22.442 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54610.log 2023-10-20T10:40:22.442 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48685.log.gz 2023-10-20T10:40:22.442 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47416.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47416.log.gz 2023-10-20T10:40:22.443 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46876.log 2023-10-20T10:40:22.443 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40496.log 2023-10-20T10:40:22.443 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54610.log.gz 2023-10-20T10:40:22.444 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41036.log 2023-10-20T10:40:22.444 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46876.log.gz 2023-10-20T10:40:22.444 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40496.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40496.log.gz 2023-10-20T10:40:22.445 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53417.log 2023-10-20T10:40:22.445 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47365.log 2023-10-20T10:40:22.445 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41036.log.gz 2023-10-20T10:40:22.445 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53417.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-20T10:40:22.446 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53417.log.gz 2023-10-20T10:40:22.446 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39048.log 2023-10-20T10:40:22.446 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47365.log.gz 2023-10-20T10:40:22.446 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51045.log 2023-10-20T10:40:22.447 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.39048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46654.log 2023-10-20T10:40:22.447 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39048.log.gz 2023-10-20T10:40:22.447 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51045.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.40658.log -- replaced with /var/log/ceph/ceph-client.admin.51045.log.gz 2023-10-20T10:40:22.448 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-20T10:40:22.448 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47389.log 2023-10-20T10:40:22.448 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46654.log.gz 2023-10-20T10:40:22.448 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43632.log 2023-10-20T10:40:22.449 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40658.log.gz 2023-10-20T10:40:22.449 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47094.log 2023-10-20T10:40:22.449 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47389.log.gz 2023-10-20T10:40:22.449 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50679.log 2023-10-20T10:40:22.450 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43632.log.gz 2023-10-20T10:40:22.450 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46498.log 2023-10-20T10:40:22.450 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47094.log.gz 2023-10-20T10:40:22.451 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43505.log 2023-10-20T10:40:22.451 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50679.log.gz 2023-10-20T10:40:22.451 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39524.log 2023-10-20T10:40:22.451 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46498.log.gz 2023-10-20T10:40:22.452 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41053.log 2023-10-20T10:40:22.452 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43505.log.gz 2023-10-20T10:40:22.452 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.39524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47539.log 2023-10-20T10:40:22.453 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39524.log.gz 2023-10-20T10:40:22.453 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47509.log 2023-10-20T10:40:22.453 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41053.log.gz 2023-10-20T10:40:22.453 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48990.log 2023-10-20T10:40:22.454 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47539.log.gz 2023-10-20T10:40:22.454 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39109.log 2023-10-20T10:40:22.454 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47509.log.gz 2023-10-20T10:40:22.455 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45370.log 2023-10-20T10:40:22.455 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48990.log.gz 2023-10-20T10:40:22.455 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.39109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54286.log 2023-10-20T10:40:22.455 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39109.log.gz 2023-10-20T10:40:22.456 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55272.log 2023-10-20T10:40:22.456 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45370.log.gz 2023-10-20T10:40:22.456 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54286.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54286.log.gzgzip 2023-10-20T10:40:22.457 INFO:teuthology.orchestra.run.smithi150.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.56016.log 2023-10-20T10:40:22.457 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56475.log 2023-10-20T10:40:22.457 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55272.log.gz 2023-10-20T10:40:22.457 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55591.log 2023-10-20T10:40:22.458 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56016.log.gz 2023-10-20T10:40:22.458 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51715.log 2023-10-20T10:40:22.458 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56475.log.gz 2023-10-20T10:40:22.459 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50528.log 2023-10-20T10:40:22.459 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55591.log.gz 2023-10-20T10:40:22.459 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41800.log 2023-10-20T10:40:22.459 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51715.log.gz 2023-10-20T10:40:22.460 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53386.log 2023-10-20T10:40:22.460 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50528.log.gz 2023-10-20T10:40:22.460 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52291.log 2023-10-20T10:40:22.460 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41800.log.gz 2023-10-20T10:40:22.461 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42490.log 2023-10-20T10:40:22.461 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53386.log.gz 2023-10-20T10:40:22.461 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46928.log 2023-10-20T10:40:22.462 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52291.log.gz 2023-10-20T10:40:22.462 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51197.log 2023-10-20T10:40:22.462 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42490.log.gz 2023-10-20T10:40:22.462 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49667.log 2023-10-20T10:40:22.463 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46928.log.gz 2023-10-20T10:40:22.463 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40550.log 2023-10-20T10:40:22.463 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51197.log.gz 2023-10-20T10:40:22.464 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40783.log 2023-10-20T10:40:22.464 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49667.log.gz 2023-10-20T10:40:22.464 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49580.log 2023-10-20T10:40:22.464 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40550.log.gz 2023-10-20T10:40:22.465 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40783.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.42593.log -- replaced with /var/log/ceph/ceph-client.admin.40783.log.gz 2023-10-20T10:40:22.465 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-20T10:40:22.465 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48746.log 2023-10-20T10:40:22.466 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49580.log.gz 2023-10-20T10:40:22.466 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55485.log 2023-10-20T10:40:22.466 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42593.log.gz 2023-10-20T10:40:22.467 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55938.log 2023-10-20T10:40:22.467 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48746.log.gz 2023-10-20T10:40:22.467 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43015.log 2023-10-20T10:40:22.467 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55485.log.gz 2023-10-20T10:40:22.468 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47828.log 2023-10-20T10:40:22.468 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55938.log.gz 2023-10-20T10:40:22.468 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50345.log 2023-10-20T10:40:22.469 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43015.log.gz 2023-10-20T10:40:22.469 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52930.log 2023-10-20T10:40:22.469 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47828.log.gz 2023-10-20T10:40:22.469 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50345.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50345.log.gz 2023-10-20T10:40:22.470 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54079.log 2023-10-20T10:40:22.470 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47122.log 2023-10-20T10:40:22.470 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52930.log.gz 2023-10-20T10:40:22.471 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54079.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54079.log.gz 2023-10-20T10:40:22.471 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34551.log 2023-10-20T10:40:22.471 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51836.log 2023-10-20T10:40:22.471 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47122.log.gz 2023-10-20T10:40:22.472 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.34551.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34551.log.gz 2023-10-20T10:40:22.472 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56341.log 2023-10-20T10:40:22.472 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50894.log 2023-10-20T10:40:22.472 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51836.log.gz 2023-10-20T10:40:22.473 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52321.log 2023-10-20T10:40:22.473 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56341.log.gz 2023-10-20T10:40:22.473 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51377.log 2023-10-20T10:40:22.474 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50894.log.gz 2023-10-20T10:40:22.474 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52321.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52321.log.gz 2023-10-20T10:40:22.474 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55858.log 2023-10-20T10:40:22.474 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41144.log 2023-10-20T10:40:22.475 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51377.log.gz 2023-10-20T10:40:22.475 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55168.log 2023-10-20T10:40:22.475 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55858.log.gz 2023-10-20T10:40:22.476 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43168.log 2023-10-20T10:40:22.476 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41144.log.gz 2023-10-20T10:40:22.476 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55168.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55168.log.gz 2023-10-20T10:40:22.476 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54899.log 2023-10-20T10:40:22.477 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52016.log 2023-10-20T10:40:22.477 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43168.log.gz 2023-10-20T10:40:22.477 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56151.log 2023-10-20T10:40:22.478 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54899.log.gz 2023-10-20T10:40:22.478 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52106.log 2023-10-20T10:40:22.478 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52016.log.gz 2023-10-20T10:40:22.479 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46528.log 2023-10-20T10:40:22.479 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56151.log.gz 2023-10-20T10:40:22.479 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52106.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52106.log.gz 2023-10-20T10:40:22.479 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43659.log 2023-10-20T10:40:22.480 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56177.log 2023-10-20T10:40:22.480 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46528.log.gz 2023-10-20T10:40:22.480 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43659.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43659.log.gz 2023-10-20T10:40:22.481 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47293.log 2023-10-20T10:40:22.481 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45759.log 2023-10-20T10:40:22.481 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56177.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56177.log.gz 2023-10-20T10:40:22.481 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46681.log 2023-10-20T10:40:22.482 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47293.log.gz 2023-10-20T10:40:22.482 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54231.log 2023-10-20T10:40:22.482 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45759.log.gz 2023-10-20T10:40:22.482 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52261.log 2023-10-20T10:40:22.483 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46681.log.gz 2023-10-20T10:40:22.483 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43198.log 2023-10-20T10:40:22.483 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54231.log.gz 2023-10-20T10:40:22.484 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51745.log 2023-10-20T10:40:22.484 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52261.log.gz 2023-10-20T10:40:22.484 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43198.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43198.log.gz 2023-10-20T10:40:22.484 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-20T10:40:22.485 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48879.log 2023-10-20T10:40:22.485 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51745.log.gz 2023-10-20T10:40:22.485 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44166.log 2023-10-20T10:40:22.486 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53782.log 2023-10-20T10:40:22.486 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48879.log.gz 2023-10-20T10:40:22.486 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44166.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51592.log 2023-10-20T10:40:22.486 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44166.log.gz 2023-10-20T10:40:22.487 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54558.log 2023-10-20T10:40:22.487 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53782.log.gz 2023-10-20T10:40:22.487 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54192.log 2023-10-20T10:40:22.488 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51592.log.gz 2023-10-20T10:40:22.488 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54662.log 2023-10-20T10:40:22.488 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54558.log.gz 2023-10-20T10:40:22.488 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41590.log 2023-10-20T10:40:22.489 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54192.log.gz 2023-10-20T10:40:22.489 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50077.log 2023-10-20T10:40:22.489 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54662.log.gz 2023-10-20T10:40:22.490 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44802.log 2023-10-20T10:40:22.490 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41590.log.gz 2023-10-20T10:40:22.490 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50741.log 2023-10-20T10:40:22.490 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50077.log.gz 2023-10-20T10:40:22.491 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41926.log 2023-10-20T10:40:22.491 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44802.log.gz 2023-10-20T10:40:22.491 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50741.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.50741.log.gz --verbose 2023-10-20T10:40:22.491 INFO:teuthology.orchestra.run.smithi150.stderr: -- /var/log/ceph/ceph-client.admin.40584.log 2023-10-20T10:40:22.492 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40854.log 2023-10-20T10:40:22.492 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41926.log.gz 2023-10-20T10:40:22.492 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46108.log 2023-10-20T10:40:22.493 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40584.log.gz 2023-10-20T10:40:22.493 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40854.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40854.log.gz -5 2023-10-20T10:40:22.493 INFO:teuthology.orchestra.run.smithi150.stderr: --verbose -- /var/log/ceph/ceph-client.admin.34634.log 2023-10-20T10:40:22.493 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46108.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46108.log.gz 2023-10-20T10:40:22.494 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54042.log 2023-10-20T10:40:22.494 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.34634.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.34634.log.gz 2023-10-20T10:40:22.494 INFO:teuthology.orchestra.run.smithi150.stderr: -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-20T10:40:22.495 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54101.log 2023-10-20T10:40:22.495 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54042.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54042.log.gz 2023-10-20T10:40:22.495 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42706.log 2023-10-20T10:40:22.495 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55005.log 2023-10-20T10:40:22.496 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54101.log.gz 2023-10-20T10:40:22.496 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42706.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42706.log.gz 2023-10-20T10:40:22.496 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47784.log 2023-10-20T10:40:22.497 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55005.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.55005.log.gz /var/log/ceph/ceph-client.admin.42243.log 2023-10-20T10:40:22.497 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-20T10:40:22.497 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47784.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47784.log.gz 2023-10-20T10:40:22.497 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55033.log 2023-10-20T10:40:22.498 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55564.log 2023-10-20T10:40:22.498 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42243.log.gz 2023-10-20T10:40:22.498 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42955.log 2023-10-20T10:40:22.499 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55033.log.gz 2023-10-20T10:40:22.499 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47448.log 2023-10-20T10:40:22.499 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55564.log.gz 2023-10-20T10:40:22.499 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.42955.log: /var/log/ceph/ceph-client.admin.53171.log 2023-10-20T10:40:22.500 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42955.log.gz 2023-10-20T10:40:22.500 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44601.log 2023-10-20T10:40:22.500 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47448.log.gz 2023-10-20T10:40:22.501 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49021.log 2023-10-20T10:40:22.501 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53171.log.gz 2023-10-20T10:40:22.501 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48634.log 2023-10-20T10:40:22.501 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44601.log.gz 2023-10-20T10:40:22.502 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39575.log 2023-10-20T10:40:22.502 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49021.log.gz 2023-10-20T10:40:22.502 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43443.log 2023-10-20T10:40:22.503 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48634.log.gz 2023-10-20T10:40:22.503 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.39575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42772.log 2023-10-20T10:40:22.503 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39575.log.gz 2023-10-20T10:40:22.503 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51685.log 2023-10-20T10:40:22.504 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43443.log.gz 2023-10-20T10:40:22.504 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42346.log 2023-10-20T10:40:22.504 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42772.log.gz 2023-10-20T10:40:22.504 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39160.log 2023-10-20T10:40:22.505 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51685.log.gz 2023-10-20T10:40:22.505 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54951.log 2023-10-20T10:40:22.505 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42346.log.gz 2023-10-20T10:40:22.506 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.39160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45479.log 2023-10-20T10:40:22.506 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39160.log.gz 2023-10-20T10:40:22.506 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40280.log 2023-10-20T10:40:22.506 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54951.log.gz 2023-10-20T10:40:22.507 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53141.log 2023-10-20T10:40:22.507 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45479.log.gz 2023-10-20T10:40:22.507 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42736.log 2023-10-20T10:40:22.508 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40280.log.gz 2023-10-20T10:40:22.508 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53141.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53141.log.gz 2023-10-20T10:40:22.508 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48806.log 2023-10-20T10:40:22.509 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42646.log 2023-10-20T10:40:22.509 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42736.log.gz 2023-10-20T10:40:22.509 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49093.log 2023-10-20T10:40:22.509 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48806.log.gz 2023-10-20T10:40:22.510 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40129.log 2023-10-20T10:40:22.510 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42646.log.gz 2023-10-20T10:40:22.510 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56367.log 2023-10-20T10:40:22.510 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49093.log.gz 2023-10-20T10:40:22.511 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50114.log 2023-10-20T10:40:22.511 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40129.log.gz 2023-10-20T10:40:22.511 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55698.log 2023-10-20T10:40:22.512 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56367.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56367.log.gz 2023-10-20T10:40:22.512 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53752.log 2023-10-20T10:40:22.512 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50114.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50114.log.gz 2023-10-20T10:40:22.512 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45087.log 2023-10-20T10:40:22.513 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55698.log.gz 2023-10-20T10:40:22.513 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53752.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30899.log 2023-10-20T10:40:22.513 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53752.log.gz 2023-10-20T10:40:22.514 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50435.log 2023-10-20T10:40:22.514 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45087.log.gz 2023-10-20T10:40:22.514 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph.tmp-client.admin.30899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49206.log 2023-10-20T10:40:22.514 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30899.log.gz 2023-10-20T10:40:22.515 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53632.log 2023-10-20T10:40:22.515 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50435.log.gz 2023-10-20T10:40:22.515 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49206.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.55194.log -- replaced with /var/log/ceph/ceph-client.admin.49206.log.gz 2023-10-20T10:40:22.516 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-20T10:40:22.516 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45728.log 2023-10-20T10:40:22.516 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53632.log.gz 2023-10-20T10:40:22.516 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55964.log 2023-10-20T10:40:22.517 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55194.log.gz 2023-10-20T10:40:22.517 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56608.log 2023-10-20T10:40:22.517 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45728.log.gz 2023-10-20T10:40:22.517 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42868.log 2023-10-20T10:40:22.518 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55964.log.gz 2023-10-20T10:40:22.518 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55777.log 2023-10-20T10:40:22.518 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56608.log.gz 2023-10-20T10:40:22.519 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48554.log 2023-10-20T10:40:22.519 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42868.log.gz 2023-10-20T10:40:22.519 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52507.log 2023-10-20T10:40:22.519 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55777.log.gz 2023-10-20T10:40:22.520 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45671.log 2023-10-20T10:40:22.520 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48554.log.gz 2023-10-20T10:40:22.520 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46064.log 2023-10-20T10:40:22.520 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52507.log.gz 2023-10-20T10:40:22.521 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52201.log 2023-10-20T10:40:22.521 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45671.log.gz 2023-10-20T10:40:22.521 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50864.log 2023-10-20T10:40:22.522 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46064.log.gz 2023-10-20T10:40:22.522 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53876.log 2023-10-20T10:40:22.522 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52201.log.gz 2023-10-20T10:40:22.522 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53906.log 2023-10-20T10:40:22.523 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50864.log.gz 2023-10-20T10:40:22.523 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44436.log 2023-10-20T10:40:22.523 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53876.log: /var/log/ceph/ceph-client.admin.53906.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53876.log.gz 2023-10-20T10:40:22.523 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51437.log 2023-10-20T10:40:22.524 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53906.log.gz 2023-10-20T10:40:22.524 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54061.log 2023-10-20T10:40:22.524 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44436.log.gz 2023-10-20T10:40:22.525 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48378.log 2023-10-20T10:40:22.525 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51437.log.gz 2023-10-20T10:40:22.525 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54847.log 2023-10-20T10:40:22.525 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54061.log.gz 2023-10-20T10:40:22.526 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47020.log 2023-10-20T10:40:22.526 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48378.log.gz 2023-10-20T10:40:22.526 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43938.log 2023-10-20T10:40:22.527 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54847.log.gz 2023-10-20T10:40:22.527 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53541.log 2023-10-20T10:40:22.527 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47020.log.gz 2023-10-20T10:40:22.527 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50984.log 2023-10-20T10:40:22.528 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43938.log.gz 2023-10-20T10:40:22.528 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42213.log 2023-10-20T10:40:22.528 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53541.log.gz 2023-10-20T10:40:22.529 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51347.log 2023-10-20T10:40:22.529 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50984.log.gz 2023-10-20T10:40:22.529 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51866.log 2023-10-20T10:40:22.529 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42213.log.gz 2023-10-20T10:40:22.530 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39186.log 2023-10-20T10:40:22.530 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51347.log.gz 2023-10-20T10:40:22.530 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51866.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.51866.log.gz /var/log/ceph/ceph-client.admin.48903.log 2023-10-20T10:40:22.531 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-20T10:40:22.531 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.39186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51986.log 2023-10-20T10:40:22.531 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39186.log.gz 2023-10-20T10:40:22.531 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50224.log 2023-10-20T10:40:22.532 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48903.log.gz 2023-10-20T10:40:22.532 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56258.log 2023-10-20T10:40:22.532 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51986.log.gz 2023-10-20T10:40:22.533 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48658.log 2023-10-20T10:40:22.533 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50224.log.gz 2023-10-20T10:40:22.533 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47263.log 2023-10-20T10:40:22.533 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56258.log.gz 2023-10-20T10:40:22.534 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48658.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48658.log.gz 2023-10-20T10:40:22.534 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45789.log 2023-10-20T10:40:22.534 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46558.log 2023-10-20T10:40:22.535 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47263.log.gz 2023-10-20T10:40:22.535 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45789.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45789.log.gz 2023-10-20T10:40:22.535 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34599.log 2023-10-20T10:40:22.535 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43968.log 2023-10-20T10:40:22.536 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46558.log.gz 2023-10-20T10:40:22.536 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.34599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44629.log 2023-10-20T10:40:22.536 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34599.log.gz 2023-10-20T10:40:22.537 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46602.log 2023-10-20T10:40:22.537 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43968.log.gz 2023-10-20T10:40:22.537 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55910.log 2023-10-20T10:40:22.537 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44629.log.gz 2023-10-20T10:40:22.538 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46602.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46602.log.gz 2023-10-20T10:40:22.538 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45509.log 2023-10-20T10:40:22.538 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41852.log 2023-10-20T10:40:22.538 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55910.log.gz 2023-10-20T10:40:22.539 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45398.log 2023-10-20T10:40:22.539 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45509.log.gz 2023-10-20T10:40:22.539 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41852.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.43228.log 2023-10-20T10:40:22.540 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41852.log.gz 2023-10-20T10:40:22.540 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42183.log 2023-10-20T10:40:22.540 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45398.log.gz 2023-10-20T10:40:22.541 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43228.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43228.log.gz 2023-10-20T10:40:22.541 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44711.log 2023-10-20T10:40:22.541 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44409.log 2023-10-20T10:40:22.541 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42183.log.gz 2023-10-20T10:40:22.542 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44711.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44711.log.gzgzip 2023-10-20T10:40:22.542 INFO:teuthology.orchestra.run.smithi150.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49694.log 2023-10-20T10:40:22.542 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53292.log 2023-10-20T10:40:22.542 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44409.log.gz 2023-10-20T10:40:22.543 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47146.log 2023-10-20T10:40:22.543 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49694.log.gz 2023-10-20T10:40:22.543 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53934.log 2023-10-20T10:40:22.544 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53292.log.gz 2023-10-20T10:40:22.544 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40766.log 2023-10-20T10:40:22.544 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47146.log.gz 2023-10-20T10:40:22.544 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45325.log 2023-10-20T10:40:22.545 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53934.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53934.log.gz 2023-10-20T10:40:22.545 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43782.log 2023-10-20T10:40:22.545 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40766.log.gz 2023-10-20T10:40:22.546 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48214.log 2023-10-20T10:40:22.546 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45325.log.gz 2023-10-20T10:40:22.546 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45232.log 2023-10-20T10:40:22.546 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43782.log.gz 2023-10-20T10:40:22.547 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48214.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48214.log.gz 2023-10-20T10:40:22.547 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52628.log 2023-10-20T10:40:22.547 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39420.log 2023-10-20T10:40:22.547 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45232.log.gz 2023-10-20T10:40:22.548 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54873.log 2023-10-20T10:40:22.548 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52628.log.gz 2023-10-20T10:40:22.548 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.39420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43752.log 2023-10-20T10:40:22.549 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39420.log.gz 2023-10-20T10:40:22.549 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51654.log 2023-10-20T10:40:22.549 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54873.log.gz 2023-10-20T10:40:22.549 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43535.log 2023-10-20T10:40:22.550 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43752.log.gz 2023-10-20T10:40:22.550 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53662.log 2023-10-20T10:40:22.550 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51654.log.gz 2023-10-20T10:40:22.551 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47203.log 2023-10-20T10:40:22.551 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43535.log.gz 2023-10-20T10:40:22.551 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53662.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53662.log.gz 2023-10-20T10:40:22.551 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54715.log 2023-10-20T10:40:22.552 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40692.log 2023-10-20T10:40:22.552 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47203.log.gz 2023-10-20T10:40:22.552 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49893.log 0.0% 2023-10-20T10:40:22.553 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54715.log.gz 2023-10-20T10:40:22.553 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50649.log 2023-10-20T10:40:22.553 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40692.log.gz 2023-10-20T10:40:22.553 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44058.log 2023-10-20T10:40:22.554 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49893.log.gz 2023-10-20T10:40:22.554 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44955.log 2023-10-20T10:40:22.554 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50649.log.gz 2023-10-20T10:40:22.554 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48930.log 2023-10-20T10:40:22.555 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44058.log.gz 2023-10-20T10:40:22.555 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51926.log 2023-10-20T10:40:22.555 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44955.log.gz 2023-10-20T10:40:22.556 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47663.log 2023-10-20T10:40:22.556 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48930.log.gz 2023-10-20T10:40:22.556 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52443.log 2023-10-20T10:40:22.556 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51926.log.gz 2023-10-20T10:40:22.557 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54361.log 2023-10-20T10:40:22.557 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47663.log.gz 2023-10-20T10:40:22.557 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44985.log 2023-10-20T10:40:22.558 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52443.log.gz 2023-10-20T10:40:22.558 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34696.log 2023-10-20T10:40:22.558 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54361.log.gz 2023-10-20T10:40:22.558 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40567.log 2023-10-20T10:40:22.559 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44985.log.gz 2023-10-20T10:40:22.559 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.34696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42430.log 2023-10-20T10:40:22.559 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34696.log.gz 2023-10-20T10:40:22.560 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42273.log 2023-10-20T10:40:22.560 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40567.log.gz 2023-10-20T10:40:22.560 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43580.log 2023-10-20T10:40:22.560 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42430.log.gz 2023-10-20T10:40:22.561 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52689.log 2023-10-20T10:40:22.561 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42273.log.gz 2023-10-20T10:40:22.561 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48462.log 2023-10-20T10:40:22.562 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43580.log.gz 2023-10-20T10:40:22.562 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52809.log 2023-10-20T10:40:22.562 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52689.log.gz 2023-10-20T10:40:22.562 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52749.log 2023-10-20T10:40:22.563 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48462.log.gz 2023-10-20T10:40:22.563 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42565.log 2023-10-20T10:40:22.563 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52809.log.gz 2023-10-20T10:40:22.564 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53262.log 2023-10-20T10:40:22.564 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52749.log.gz 2023-10-20T10:40:22.564 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45944.log 2023-10-20T10:40:22.564 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42565.log.gz 2023-10-20T10:40:22.565 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48599.log 2023-10-20T10:40:22.565 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53262.log.gz 2023-10-20T10:40:22.565 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40604.log 2023-10-20T10:40:22.565 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45944.log.gz 2023-10-20T10:40:22.566 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56661.log 2023-10-20T10:40:22.566 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48599.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48599.log.gz 2023-10-20T10:40:22.566 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40459.log 2023-10-20T10:40:22.567 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40604.log.gz 2023-10-20T10:40:22.567 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48402.log 2023-10-20T10:40:22.567 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56661.log.gz 2023-10-20T10:40:22.567 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45119.log 2023-10-20T10:40:22.568 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40459.log.gz 2023-10-20T10:40:22.568 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48402.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.48402.log.gz -- 2023-10-20T10:40:22.568 INFO:teuthology.orchestra.run.smithi150.stderr: /var/log/ceph/ceph-client.admin.56068.log 2023-10-20T10:40:22.569 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42074.log 2023-10-20T10:40:22.569 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45119.log.gz 2023-10-20T10:40:22.569 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45893.log 2023-10-20T10:40:22.569 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56068.log.gz 2023-10-20T10:40:22.570 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53449.log 2023-10-20T10:40:22.570 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42074.log.gz 2023-10-20T10:40:22.570 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5/var/log/ceph/ceph-client.admin.45893.log: --verbose -- /var/log/ceph/ceph-client.admin.56580.log 2023-10-20T10:40:22.571 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45893.log.gz 2023-10-20T10:40:22.571 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40422.log 2023-10-20T10:40:22.571 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53449.log.gz 2023-10-20T10:40:22.572 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45698.log 2023-10-20T10:40:22.572 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56580.log.gz 2023-10-20T10:40:22.572 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54138.log 2023-10-20T10:40:22.573 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40422.log.gz 2023-10-20T10:40:22.573 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53970.log 2023-10-20T10:40:22.573 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45698.log.gz 2023-10-20T10:40:22.574 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41506.log 2023-10-20T10:40:22.574 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54138.log.gz 2023-10-20T10:40:22.574 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40388.log 2023-10-20T10:40:22.574 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53970.log.gz 2023-10-20T10:40:22.575 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48184.log 2023-10-20T10:40:22.575 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41506.log.gz 2023-10-20T10:40:22.575 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50498.log 2023-10-20T10:40:22.576 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40388.log.gz 2023-10-20T10:40:22.576 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55142.log 2023-10-20T10:40:22.576 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48184.log.gz 2023-10-20T10:40:22.577 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53692.log 2023-10-20T10:40:22.577 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50498.log.gz 2023-10-20T10:40:22.577 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39424.log 2023-10-20T10:40:22.577 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55142.log.gz 2023-10-20T10:40:22.578 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54006.log 2023-10-20T10:40:22.578 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53692.log.gz 2023-10-20T10:40:22.578 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.39424.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39424.log.gz -5 2023-10-20T10:40:22.579 INFO:teuthology.orchestra.run.smithi150.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50284.log 2023-10-20T10:40:22.579 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40203.log 2023-10-20T10:40:22.579 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54006.log.gz 2023-10-20T10:40:22.579 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43117.log 2023-10-20T10:40:22.580 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50284.log.gz 2023-10-20T10:40:22.580 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40203.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40203.log.gz 2023-10-20T10:40:22.580 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51407.log 2023-10-20T10:40:22.581 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40334.log 2023-10-20T10:40:22.581 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43117.log.gz 2023-10-20T10:40:22.581 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49725.log 2023-10-20T10:40:22.581 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51407.log.gz 2023-10-20T10:40:22.582 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41978.log 2023-10-20T10:40:22.582 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40334.log.gz 2023-10-20T10:40:22.582 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53988.log 2023-10-20T10:40:22.583 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49725.log.gz 2023-10-20T10:40:22.583 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-20T10:40:22.583 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41978.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41978.log.gz 2023-10-20T10:40:22.583 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46136.log 2023-10-20T10:40:22.584 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53988.log.gz 2023-10-20T10:40:22.584 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53511.log 2023-10-20T10:40:22.584 INFO:teuthology.orchestra.run.smithi150.stderr:gzip/var/log/ceph/ceph-client.admin.46136.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47938.log 2023-10-20T10:40:22.585 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46136.log.gz 2023-10-20T10:40:22.585 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50618.log 2023-10-20T10:40:22.585 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53511.log.gz 2023-10-20T10:40:22.585 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41107.log 2023-10-20T10:40:22.586 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47938.log.gz 2023-10-20T10:40:22.586 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51015.log 2023-10-20T10:40:22.586 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50618.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50618.log.gz 2023-10-20T10:40:22.587 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41107.log.gz 2023-10-20T10:40:22.587 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53952.log 2023-10-20T10:40:22.587 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42844.log 2023-10-20T10:40:22.588 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51015.log.gz 2023-10-20T10:40:22.588 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40368.log 2023-10-20T10:40:22.588 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53952.log.gz 2023-10-20T10:40:22.588 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45974.log 2023-10-20T10:40:22.589 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42844.log.gz 2023-10-20T10:40:22.589 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40982.log 2023-10-20T10:40:22.589 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40368.log.gz 2023-10-20T10:40:22.590 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45974.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45974.log.gz 2023-10-20T10:40:22.590 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49615.log 2023-10-20T10:40:22.590 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43288.log 2023-10-20T10:40:22.591 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40982.log.gz 2023-10-20T10:40:22.591 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53325.log 2023-10-20T10:40:22.591 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49615.log.gz 2023-10-20T10:40:22.591 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45570.log 2023-10-20T10:40:22.592 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43288.log.gz 2023-10-20T10:40:22.592 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54428.log 2023-10-20T10:40:22.592 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53325.log.gz 2023-10-20T10:40:22.592 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54978.log 2023-10-20T10:40:22.593 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45570.log.gz 2023-10-20T10:40:22.593 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40729.log 2023-10-20T10:40:22.593 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54428.log.gz 2023-10-20T10:40:22.593 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56715.log 2023-10-20T10:40:22.594 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54978.log.gz 2023-10-20T10:40:22.594 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40351.log 2023-10-20T10:40:22.594 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40729.log.gz 2023-10-20T10:40:22.595 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41124.log 2023-10-20T10:40:22.595 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56715.log.gz 2023-10-20T10:40:22.595 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47337.log 2023-10-20T10:40:22.595 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40351.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40351.log.gz 2023-10-20T10:40:22.596 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44028.log 2023-10-20T10:40:22.596 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41124.log.gz 2023-10-20T10:40:22.596 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53081.log 2023-10-20T10:40:22.596 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47337.log.gz 2023-10-20T10:40:22.597 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55832.log 2023-10-20T10:40:22.597 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44028.log.gz 2023-10-20T10:40:22.597 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48432.log 2023-10-20T10:40:22.598 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53081.log.gz 2023-10-20T10:40:22.598 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45144.log 2023-10-20T10:40:22.598 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55832.log.gz 2023-10-20T10:40:22.599 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48133.log 2023-10-20T10:40:22.599 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48432.log.gz 2023-10-20T10:40:22.599 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51956.log 2023-10-20T10:40:22.600 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45144.log.gz 2023-10-20T10:40:22.600 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55803.log 0.0% 2023-10-20T10:40:22.600 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48133.log.gz 2023-10-20T10:40:22.600 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44223.log 2023-10-20T10:40:22.601 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51956.log.gz 2023-10-20T10:40:22.601 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46803.log 2023-10-20T10:40:22.601 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55803.log.gz 2023-10-20T10:40:22.602 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44223.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44223.log.gz 2023-10-20T10:40:22.602 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42925.log 2023-10-20T10:40:22.602 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43362.log 2023-10-20T10:40:22.602 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46803.log.gz 2023-10-20T10:40:22.603 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40476.log 2023-10-20T10:40:22.603 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42925.log.gz 2023-10-20T10:40:22.603 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54454.log 2023-10-20T10:40:22.604 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43362.log.gz 2023-10-20T10:40:22.604 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43690.log 2023-10-20T10:40:22.604 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40476.log.gz 2023-10-20T10:40:22.604 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41216.log 2023-10-20T10:40:22.605 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54454.log.gz 2023-10-20T10:40:22.605 INFO:teuthology.orchestra.run.smithi150.stderr:gzip/var/log/ceph/ceph-client.admin.43690.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51136.log 2023-10-20T10:40:22.605 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43690.log.gz 2023-10-20T10:40:22.606 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44193.log 2023-10-20T10:40:22.606 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41216.log.gz 2023-10-20T10:40:22.606 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52382.log 2023-10-20T10:40:22.607 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51136.log.gz 2023-10-20T10:40:22.607 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48492.log/var/log/ceph/ceph-client.admin.44193.log: 2023-10-20T10:40:22.607 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44193.log.gz 2023-10-20T10:40:22.607 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49643.log 2023-10-20T10:40:22.608 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52382.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52382.log.gz 2023-10-20T10:40:22.608 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42520.log 2023-10-20T10:40:22.608 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48492.log.gz 2023-10-20T10:40:22.609 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46900.log 2023-10-20T10:40:22.609 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49643.log.gz 2023-10-20T10:40:22.609 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50034.log 2023-10-20T10:40:22.610 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42520.log.gz 2023-10-20T10:40:22.610 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46900.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46900.log.gz 2023-10-20T10:40:22.610 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43905.log 2023-10-20T10:40:22.610 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51317.log 2023-10-20T10:40:22.611 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50034.log.gz 2023-10-20T10:40:22.611 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39271.log 2023-10-20T10:40:22.611 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43905.log.gz 2023-10-20T10:40:22.611 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44653.log 2023-10-20T10:40:22.612 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51317.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51317.log.gz 2023-10-20T10:40:22.612 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.39271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52900.log 2023-10-20T10:40:22.612 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39271.log.gz 2023-10-20T10:40:22.613 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55407.log 2023-10-20T10:40:22.613 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44653.log.gz 2023-10-20T10:40:22.613 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51529.log 2023-10-20T10:40:22.614 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52900.log.gz 2023-10-20T10:40:22.614 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55328.log 2023-10-20T10:40:22.614 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55407.log.gz 2023-10-20T10:40:22.615 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54480.log 2023-10-20T10:40:22.615 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51529.log.gz 2023-10-20T10:40:22.615 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44385.log 2023-10-20T10:40:22.615 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55328.log.gz 2023-10-20T10:40:22.616 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40442.log 2023-10-20T10:40:22.616 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54480.log.gz 2023-10-20T10:40:22.616 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52658.log 2023-10-20T10:40:22.617 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44385.log.gz 2023-10-20T10:40:22.617 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46407.log 2023-10-20T10:40:22.617 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40442.log.gz 2023-10-20T10:40:22.617 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42460.log 2023-10-20T10:40:22.618 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52658.log.gz 2023-10-20T10:40:22.618 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46004.log 2023-10-20T10:40:22.618 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46407.log.gz 2023-10-20T10:40:22.619 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41548.log 2023-10-20T10:40:22.619 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42460.log.gz 2023-10-20T10:40:22.619 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53479.log 2023-10-20T10:40:22.619 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46004.log.gz 2023-10-20T10:40:22.620 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51166.log 2023-10-20T10:40:22.620 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41548.log.gz 2023-10-20T10:40:22.620 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42676.log 2023-10-20T10:40:22.621 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53479.log.gz 2023-10-20T10:40:22.621 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45821.log 2023-10-20T10:40:22.621 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51166.log.gz 2023-10-20T10:40:22.621 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49155.log 2023-10-20T10:40:22.622 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42676.log.gz 2023-10-20T10:40:22.622 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45821.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-20T10:40:22.622 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45821.log.gz 2023-10-20T10:40:22.623 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42370.log 2023-10-20T10:40:22.623 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49155.log.gz 2023-10-20T10:40:22.623 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54024.log 2023-10-20T10:40:22.623 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56739.log 2023-10-20T10:40:22.624 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42370.log.gz 2023-10-20T10:40:22.624 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44898.log 2023-10-20T10:40:22.624 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54024.log.gz 2023-10-20T10:40:22.625 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55990.log 2023-10-20T10:40:22.625 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56739.log.gz 2023-10-20T10:40:22.625 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54250.log 2023-10-20T10:40:22.625 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44898.log.gz 2023-10-20T10:40:22.626 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49369.log 2023-10-20T10:40:22.626 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55990.log.gz 2023-10-20T10:40:22.626 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52960.log 2023-10-20T10:40:22.627 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54250.log.gz 2023-10-20T10:40:22.627 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49369.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-20T10:40:22.627 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49369.log.gz 2023-10-20T10:40:22.628 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43878.log 2023-10-20T10:40:22.628 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52960.log.gz 2023-10-20T10:40:22.628 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45619.log 2023-10-20T10:40:22.628 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43878.log.gz 2023-10-20T10:40:22.629 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40908.log 2023-10-20T10:40:22.629 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41758.log 2023-10-20T10:40:22.629 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45619.log.gz 2023-10-20T10:40:22.629 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40908.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40908.log.gz 2023-10-20T10:40:22.630 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52598.log 2023-10-20T10:40:22.630 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44874.log 2023-10-20T10:40:22.630 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41758.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41758.log.gz 2023-10-20T10:40:22.631 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39272.log 2023-10-20T10:40:22.631 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42318.log 2023-10-20T10:40:22.631 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44874.log.gz 2023-10-20T10:40:22.631 INFO:teuthology.orchestra.run.smithi150.stderr: 28.0%/var/log/ceph/ceph-client.admin.39272.log: -- replaced with /var/log/ceph/ceph-client.admin.52598.log.gz 2023-10-20T10:40:22.632 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40746.log 2023-10-20T10:40:22.632 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39272.log.gz 2023-10-20T10:40:22.632 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42318.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42318.log.gz 2023-10-20T10:40:22.633 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54402.log 2023-10-20T10:40:22.633 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41016.log 2023-10-20T10:40:22.633 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40746.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40746.log.gz 2023-10-20T10:40:22.633 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52538.log 2023-10-20T10:40:22.634 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54402.log.gz 2023-10-20T10:40:22.634 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55246.log 2023-10-20T10:40:22.634 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41016.log.gz 2023-10-20T10:40:22.635 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43826.log 2023-10-20T10:40:22.635 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52538.log.gz 2023-10-20T10:40:22.635 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39212.log 2023-10-20T10:40:22.635 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55246.log.gz 2023-10-20T10:40:22.636 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41726.log 2023-10-20T10:40:22.636 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43826.log.gz 2023-10-20T10:40:22.636 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.39212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39342.log 2023-10-20T10:40:22.636 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39212.log.gz 2023-10-20T10:40:22.637 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47584.log 2023-10-20T10:40:22.637 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41726.log.gz 2023-10-20T10:40:22.637 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.39342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47173.log 2023-10-20T10:40:22.637 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39342.log.gz 2023-10-20T10:40:22.638 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47584.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47584.log.gz 2023-10-20T10:40:22.638 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50465.log 2023-10-20T10:40:22.638 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49393.log 2023-10-20T10:40:22.639 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47173.log.gz 2023-10-20T10:40:22.639 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53356.log 2023-10-20T10:40:22.639 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50465.log.gz 2023-10-20T10:40:22.639 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49393.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49393.log.gz 2023-10-20T10:40:22.640 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41422.log 2023-10-20T10:40:22.640 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42397.log 2023-10-20T10:40:22.640 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53356.log.gz 2023-10-20T10:40:22.641 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54636.log 2023-10-20T10:40:22.641 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41422.log.gz 2023-10-20T10:40:22.641 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42397.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42397.log.gz 2023-10-20T10:40:22.641 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44111.log 2023-10-20T10:40:22.642 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48275.log 2023-10-20T10:40:22.642 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54636.log.gz 2023-10-20T10:40:22.642 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49127.log 2023-10-20T10:40:22.642 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44111.log.gz 2023-10-20T10:40:22.643 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42812.log 2023-10-20T10:40:22.643 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48275.log.gz 2023-10-20T10:40:22.643 INFO:teuthology.orchestra.run.smithi150.stderr: 89.7% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-20T10:40:22.644 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55355.log 2023-10-20T10:40:22.644 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49127.log.gz 2023-10-20T10:40:22.644 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5/var/log/ceph/ceph-client.admin.42812.log: --verbose -- /var/log/ceph/ceph-client.admin.54584.log 2023-10-20T10:40:22.644 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42812.log.gz 2023-10-20T10:40:22.645 INFO:teuthology.orchestra.run.smithi150.stderr:gzip/var/log/ceph/ceph-client.admin.55355.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47233.log 2023-10-20T10:40:22.645 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55355.log.gz 2023-10-20T10:40:22.645 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45264.log 2023-10-20T10:40:22.645 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54584.log.gz 2023-10-20T10:40:22.646 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39134.log 2023-10-20T10:40:22.646 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47233.log.gz 2023-10-20T10:40:22.646 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52475.log 2023-10-20T10:40:22.647 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45264.log.gz 2023-10-20T10:40:22.647 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.39134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51287.log 2023-10-20T10:40:22.647 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39134.log.gz 2023-10-20T10:40:22.647 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49540.log 2023-10-20T10:40:22.648 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52475.log.gz 2023-10-20T10:40:22.648 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54119.log 2023-10-20T10:40:22.648 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51287.log.gz 2023-10-20T10:40:22.648 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48776.log 2023-10-20T10:40:22.649 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49540.log.gz 2023-10-20T10:40:22.649 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41178.log 2023-10-20T10:40:22.649 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54119.log.gz 2023-10-20T10:40:22.650 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45449.log 2023-10-20T10:40:22.650 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48776.log.gz 2023-10-20T10:40:22.650 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54211.log 2023-10-20T10:40:22.650 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41178.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41178.log.gz 2023-10-20T10:40:22.651 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51559.log 2023-10-20T10:40:22.651 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45449.log.gz 2023-10-20T10:40:22.651 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51467.log 2023-10-20T10:40:22.652 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54211.log.gz 2023-10-20T10:40:22.652 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54324.log 2023-10-20T10:40:22.652 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51559.log.gz 2023-10-20T10:40:22.652 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54768.log 2023-10-20T10:40:22.653 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51467.log.gz 2023-10-20T10:40:22.653 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40226.log 2023-10-20T10:40:22.653 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54324.log.gz 2023-10-20T10:40:22.654 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50558.log 2023-10-20T10:40:22.654 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54768.log.gz 2023-10-20T10:40:22.654 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52991.log 2023-10-20T10:40:22.654 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40226.log.gz 2023-10-20T10:40:22.655 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50558.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.50558.log.gz -- 2023-10-20T10:40:22.655 INFO:teuthology.orchestra.run.smithi150.stderr: /var/log/ceph/ceph-client.admin.34457.log 2023-10-20T10:40:22.655 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56230.log 2023-10-20T10:40:22.656 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52991.log.gz 2023-10-20T10:40:22.656 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.34457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49865.log 2023-10-20T10:40:22.656 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34457.log.gz 2023-10-20T10:40:22.656 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45539.log 2023-10-20T10:40:22.657 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56230.log.gz 2023-10-20T10:40:22.657 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48030.log 2023-10-20T10:40:22.657 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49865.log.gz 2023-10-20T10:40:22.658 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50142.log 2023-10-20T10:40:22.658 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45539.log.gz 2023-10-20T10:40:22.658 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47858.log 2023-10-20T10:40:22.658 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48030.log.gz 2023-10-20T10:40:22.659 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55646.log 2023-10-20T10:40:22.659 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50142.log.gz 2023-10-20T10:40:22.659 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53845.log 2023-10-20T10:40:22.659 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47858.log.gz 2023-10-20T10:40:22.660 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55116.log 2023-10-20T10:40:22.660 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55646.log.gz 2023-10-20T10:40:22.660 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55511.log 2023-10-20T10:40:22.661 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53845.log.gz 2023-10-20T10:40:22.661 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49917.log 2023-10-20T10:40:22.661 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55116.log.gz 2023-10-20T10:40:22.661 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43854.log 2023-10-20T10:40:22.662 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55511.log.gz 2023-10-20T10:40:22.662 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39238.log 2023-10-20T10:40:22.662 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49917.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49917.log.gz 2023-10-20T10:40:22.662 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42617.log 2023-10-20T10:40:22.663 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43854.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43854.log.gz 2023-10-20T10:40:22.663 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.39238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41090.log 2023-10-20T10:40:22.663 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39238.log.gz 2023-10-20T10:40:22.664 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40405.log 2023-10-20T10:40:22.664 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42617.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42617.log.gz 2023-10-20T10:40:22.664 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50771.log 2023-10-20T10:40:22.664 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41090.log.gz 2023-10-20T10:40:22.665 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40405.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40405.log.gz 2023-10-20T10:40:22.665 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39428.log 2023-10-20T10:40:22.665 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41632.log 2023-10-20T10:40:22.665 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50771.log.gz 2023-10-20T10:40:22.666 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.39428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48350.log 2023-10-20T10:40:22.666 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39428.log.gz 2023-10-20T10:40:22.666 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50803.log 2023-10-20T10:40:22.667 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41632.log.gz 2023-10-20T10:40:22.667 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.48350.log: /var/log/ceph/ceph-client.admin.45647.log 2023-10-20T10:40:22.667 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48350.log.gz 2023-10-20T10:40:22.667 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50803.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50803.log.gz -5 2023-10-20T10:40:22.668 INFO:teuthology.orchestra.run.smithi150.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39498.log 2023-10-20T10:40:22.668 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40186.log 2023-10-20T10:40:22.668 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45647.log.gz 2023-10-20T10:40:22.668 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.39498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40297.log 2023-10-20T10:40:22.669 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39498.log.gz 2023-10-20T10:40:22.669 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40186.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40186.log.gz 2023-10-20T10:40:22.669 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51257.log 2023-10-20T10:40:22.670 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40297.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40297.log.gz 2023-10-20T10:40:22.670 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46217.log 2023-10-20T10:40:22.670 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49179.log 2023-10-20T10:40:22.670 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51257.log.gz 2023-10-20T10:40:22.671 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51498.log 2023-10-20T10:40:22.671 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46217.log.gz 2023-10-20T10:40:22.671 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40243.log 2023-10-20T10:40:22.672 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49179.log.gz 2023-10-20T10:40:22.672 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52412.log 2023-10-20T10:40:22.672 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51498.log.gz 2023-10-20T10:40:22.672 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47969.log 2023-10-20T10:40:22.673 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40243.log.gz 2023-10-20T10:40:22.673 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55220.log 2023-10-20T10:40:22.673 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52412.log.gz 2023-10-20T10:40:22.673 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48523.log 2023-10-20T10:40:22.674 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47969.log.gz 2023-10-20T10:40:22.674 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50954.log 2023-10-20T10:40:22.674 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55220.log.gz 2023-10-20T10:40:22.674 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46711.log 2023-10-20T10:40:22.675 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48523.log.gz 2023-10-20T10:40:22.675 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40945.log 2023-10-20T10:40:22.675 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50954.log.gz 2023-10-20T10:40:22.676 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.46711.log: -- /var/log/ceph/ceph-client.admin.43608.log 2023-10-20T10:40:22.676 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46711.log.gz 2023-10-20T10:40:22.676 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50588.log 2023-10-20T10:40:22.676 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40945.log.gz 2023-10-20T10:40:22.677 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49297.log 2023-10-20T10:40:22.677 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43608.log.gz 2023-10-20T10:40:22.677 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50588.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-20T10:40:22.677 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50588.log.gz 2023-10-20T10:40:22.678 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49297.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49297.log.gz 2023-10-20T10:40:22.678 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54925.log 2023-10-20T10:40:22.678 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49944.log 2023-10-20T10:40:22.679 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56395.log 2023-10-20T10:40:22.679 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54925.log.gz 2023-10-20T10:40:22.679 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55301.log 2023-10-20T10:40:22.679 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49944.log.gz 2023-10-20T10:40:22.680 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41884.log 2023-10-20T10:40:22.680 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56395.log.gz 2023-10-20T10:40:22.680 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46311.log 2023-10-20T10:40:22.681 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55301.log.gz 2023-10-20T10:40:22.681 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53571.log 2023-10-20T10:40:22.681 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41884.log.gz 2023-10-20T10:40:22.681 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43720.log 2023-10-20T10:40:22.682 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46311.log.gz 2023-10-20T10:40:22.682 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48716.log 2023-10-20T10:40:22.682 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53571.log.gz 2023-10-20T10:40:22.683 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52076.log 2023-10-20T10:40:22.683 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43720.log.gz 2023-10-20T10:40:22.683 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52779.log 2023-10-20T10:40:22.683 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48716.log.gz 2023-10-20T10:40:22.684 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40166.log 2023-10-20T10:40:22.684 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52076.log.gz 2023-10-20T10:40:22.684 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54532.log 2023-10-20T10:40:22.684 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52779.log.gz 2023-10-20T10:40:22.685 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46248.log 2023-10-20T10:40:22.685 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40166.log.gz 2023-10-20T10:40:22.685 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55884.log 2023-10-20T10:40:22.686 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54532.log.gz 2023-10-20T10:40:22.686 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39269.log 2023-10-20T10:40:22.686 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46248.log.gz 2023-10-20T10:40:22.686 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44925.log 2023-10-20T10:40:22.687 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55884.log.gz 2023-10-20T10:40:22.687 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.39269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52719.log 2023-10-20T10:40:22.687 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39269.log.gz 2023-10-20T10:40:22.688 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50004.log 2023-10-20T10:40:22.688 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44925.log.gz 2023-10-20T10:40:22.688 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50405.log 2023-10-20T10:40:22.688 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52719.log.gz 2023-10-20T10:40:22.689 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46468.log 2023-10-20T10:40:22.689 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50004.log.gz 2023-10-20T10:40:22.689 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41070.log 2023-10-20T10:40:22.690 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50405.log.gz 2023-10-20T10:40:22.690 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41682.log 2023-10-20T10:40:22.690 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46468.log.gz 2023-10-20T10:40:22.690 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41070.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.41070.log.gz 2023-10-20T10:40:22.691 INFO:teuthology.orchestra.run.smithi150.stderr: /var/log/ceph/ceph-client.admin.47999.log 2023-10-20T10:40:22.691 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56448.log 2023-10-20T10:40:22.691 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41682.log.gz 2023-10-20T10:40:22.692 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53051.log 2023-10-20T10:40:22.692 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47999.log.gz 2023-10-20T10:40:22.692 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41198.log 2023-10-20T10:40:22.693 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56448.log.gz 2023-10-20T10:40:22.693 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53111.log 2023-10-20T10:40:22.693 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53051.log.gz 2023-10-20T10:40:22.693 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40638.log 2023-10-20T10:40:22.694 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41198.log.gz 2023-10-20T10:40:22.694 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53111.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53111.log.gz -5 2023-10-20T10:40:22.694 INFO:teuthology.orchestra.run.smithi150.stderr: --verbose -- /var/log/ceph/ceph-client.admin.54742.log 2023-10-20T10:40:22.695 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45201.log 2023-10-20T10:40:22.695 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40638.log.gz 2023-10-20T10:40:22.695 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34575.log 2023-10-20T10:40:22.696 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54742.log.gz 2023-10-20T10:40:22.696 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45201.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.45201.log.gz -5 2023-10-20T10:40:22.696 INFO:teuthology.orchestra.run.smithi150.stderr: --verbose -- /var/log/ceph/ceph-client.admin.51076.log 2023-10-20T10:40:22.697 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.34575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50375.log 2023-10-20T10:40:22.697 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34575.log.gz 2023-10-20T10:40:22.697 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43045.log 2023-10-20T10:40:22.697 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51076.log.gz 2023-10-20T10:40:22.698 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50924.log 2023-10-20T10:40:22.698 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50375.log.gz 2023-10-20T10:40:22.698 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52351.log 2023-10-20T10:40:22.699 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43045.log.gz 2023-10-20T10:40:22.699 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43258.log 2023-10-20T10:40:22.699 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50924.log.gz 2023-10-20T10:40:22.699 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54174.log 2023-10-20T10:40:22.700 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52351.log.gz 2023-10-20T10:40:22.700 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45047.log 2023-10-20T10:40:22.700 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43258.log.gz 2023-10-20T10:40:22.701 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44846.log 2023-10-20T10:40:22.701 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54174.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54174.log.gz 2023-10-20T10:40:22.701 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43413.log 2023-10-20T10:40:22.702 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45047.log.gz 2023-10-20T10:40:22.702 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47636.log 2023-10-20T10:40:22.702 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44846.log.gz 2023-10-20T10:40:22.702 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42985.log 2023-10-20T10:40:22.703 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43413.log.gz 2023-10-20T10:40:22.703 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42102.log 2023-10-20T10:40:22.703 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47636.log.gz 2023-10-20T10:40:22.704 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39350.log 2023-10-20T10:40:22.704 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42985.log.gz 2023-10-20T10:40:22.704 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49480.log 2023-10-20T10:40:22.704 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42102.log.gz 2023-10-20T10:40:22.705 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.39350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49819.log 2023-10-20T10:40:22.705 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39350.log.gz 2023-10-20T10:40:22.705 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56096.log 2023-10-20T10:40:22.706 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49480.log.gz 2023-10-20T10:40:22.706 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40874.log 2023-10-20T10:40:22.706 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49819.log.gz 2023-10-20T10:40:22.707 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46355.log 2023-10-20T10:40:22.707 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56096.log.gz 2023-10-20T10:40:22.707 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47693.log 2023-10-20T10:40:22.707 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40874.log.gz 2023-10-20T10:40:22.708 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51775.log 2023-10-20T10:40:22.708 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46355.log.gz 2023-10-20T10:40:22.708 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56687.log 2023-10-20T10:40:22.709 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47693.log.gz 2023-10-20T10:40:22.709 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45016.log 2023-10-20T10:40:22.709 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51775.log.gz 2023-10-20T10:40:22.710 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56687.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.48105.log 2023-10-20T10:40:22.710 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56687.log.gz 2023-10-20T10:40:22.710 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41464.log 2023-10-20T10:40:22.710 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45016.log.gz 2023-10-20T10:40:22.711 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44313.log 2023-10-20T10:40:22.711 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48105.log.gz 2023-10-20T10:40:22.711 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50166.log 2023-10-20T10:40:22.712 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41464.log.gz 2023-10-20T10:40:22.712 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56314.log 2023-10-20T10:40:22.712 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44313.log.gz 2023-10-20T10:40:22.712 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56042.log 2023-10-20T10:40:22.713 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50166.log.gz 2023-10-20T10:40:22.713 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54268.log 2023-10-20T10:40:22.713 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56314.log.gz 2023-10-20T10:40:22.714 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55618.log 2023-10-20T10:40:22.714 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56042.log.gz 2023-10-20T10:40:22.714 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50709.log 2023-10-20T10:40:22.714 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54268.log.gz 2023-10-20T10:40:22.715 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48960.log 2023-10-20T10:40:22.715 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55618.log.gz 2023-10-20T10:40:22.715 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56422.log 2023-10-20T10:40:22.716 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50709.log.gz 2023-10-20T10:40:22.716 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40820.log 2023-10-20T10:40:22.716 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48960.log.gz 2023-10-20T10:40:22.716 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46990.log 2023-10-20T10:40:22.717 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56422.log.gz 2023-10-20T10:40:22.717 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44527.log 2023-10-20T10:40:22.717 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40820.log.gz 2023-10-20T10:40:22.717 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46160.log 2023-10-20T10:40:22.718 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48851.log 2023-10-20T10:40:22.718 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44527.log.gz 2023-10-20T10:40:22.718 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47886.log 2023-10-20T10:40:22.719 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46160.log.gz 2023-10-20T10:40:22.719 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47612.log 2023-10-20T10:40:22.719 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48851.log.gz 2023-10-20T10:40:22.719 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49420.log 2023-10-20T10:40:22.720 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47886.log.gz 2023-10-20T10:40:22.720 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47612.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.47612.log.gz -- 2023-10-20T10:40:22.720 INFO:teuthology.orchestra.run.smithi150.stderr: /var/log/ceph/ceph-client.admin.43998.log 2023-10-20T10:40:22.721 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40146.log 2023-10-20T10:40:22.721 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49420.log.gz 2023-10-20T10:40:22.721 INFO:teuthology.orchestra.run.smithi150.stderr:gzip/var/log/ceph/ceph-client.admin.43998.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41264.log 2023-10-20T10:40:22.721 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43998.log.gz 2023-10-20T10:40:22.722 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54689.log 2023-10-20T10:40:22.722 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40146.log.gz 2023-10-20T10:40:22.722 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53021.log 2023-10-20T10:40:22.723 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41264.log.gz 2023-10-20T10:40:22.723 INFO:teuthology.orchestra.run.smithi150.stderr: 28.5% -- replaced with /var/log/ceph/ceph-client.admin.46990.log.gz 2023-10-20T10:40:22.723 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52568.log 2023-10-20T10:40:22.723 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54689.log.gz 2023-10-20T10:40:22.724 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49757.log 2023-10-20T10:40:22.724 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53021.log.gz 2023-10-20T10:40:22.724 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49052.log 2023-10-20T10:40:22.725 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52568.log.gz 2023-10-20T10:40:22.725 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48305.log 2023-10-20T10:40:22.725 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49757.log.gz 2023-10-20T10:40:22.725 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56204.log 2023-10-20T10:40:22.726 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49052.log.gz 2023-10-20T10:40:22.726 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42126.log 2023-10-20T10:40:22.726 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48305.log.gz 2023-10-20T10:40:22.727 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46773.log 2023-10-20T10:40:22.727 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56204.log.gz 2023-10-20T10:40:22.727 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39348.log 2023-10-20T10:40:22.727 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42126.log.gz 2023-10-20T10:40:22.728 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51227.log 2023-10-20T10:40:22.728 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46773.log.gz 2023-10-20T10:40:22.728 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.39348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43334.log 2023-10-20T10:40:22.729 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39348.log.gz 2023-10-20T10:40:22.729 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42020.log 2023-10-20T10:40:22.729 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51227.log.gz 2023-10-20T10:40:22.729 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40999.log 2023-10-20T10:40:22.730 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43334.log.gz 2023-10-20T10:40:22.730 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40712.log 2023-10-20T10:40:22.730 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42020.log.gz 2023-10-20T10:40:22.731 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45294.log 2023-10-20T10:40:22.731 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40999.log.gz 2023-10-20T10:40:22.731 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44681.log 2023-10-20T10:40:22.731 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40712.log.gz 2023-10-20T10:40:22.732 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45294.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45294.log.gz 2023-10-20T10:40:22.732 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44466.log 2023-10-20T10:40:22.732 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53812.log 2023-10-20T10:40:22.732 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44681.log.gz 2023-10-20T10:40:22.733 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55672.log 2023-10-20T10:40:22.733 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44466.log.gz 2023-10-20T10:40:22.733 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49788.log 2023-10-20T10:40:22.733 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53812.log.gz 2023-10-20T10:40:22.734 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55672.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55672.log.gz 2023-10-20T10:40:22.734 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54506.log 2023-10-20T10:40:22.734 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54820.log 2023-10-20T10:40:22.735 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49788.log.gz 2023-10-20T10:40:22.735 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54506.log.gz 2023-10-20T10:40:22.735 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55381.log 2023-10-20T10:40:22.736 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.54820.log: /var/log/ceph/ceph-client.admin.56284.log 2023-10-20T10:40:22.736 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54820.log.gz 2023-10-20T10:40:22.736 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46034.log 2023-10-20T10:40:22.736 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55381.log.gz 2023-10-20T10:40:22.737 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52231.log 2023-10-20T10:40:22.737 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56284.log.gz 2023-10-20T10:40:22.737 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40891.log 2023-10-20T10:40:22.737 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46034.log.gz 2023-10-20T10:40:22.738 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48245.log 2023-10-20T10:40:22.738 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52231.log.gz 2023-10-20T10:40:22.738 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56635.log 2023-10-20T10:40:22.739 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40891.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40891.log.gz 2023-10-20T10:40:22.739 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50315.log 2023-10-20T10:40:22.739 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48245.log.gz 2023-10-20T10:40:22.739 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47911.log 2023-10-20T10:40:22.740 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56635.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56635.log.gz 2023-10-20T10:40:22.740 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49267.log 2023-10-20T10:40:22.740 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50315.log.gz 2023-10-20T10:40:22.741 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44497.log 2023-10-20T10:40:22.741 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47911.log.gz 2023-10-20T10:40:22.741 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43386.log 2023-10-20T10:40:22.741 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49267.log.gz 2023-10-20T10:40:22.742 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44497.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44497.log.gz 2023-10-20T10:40:22.742 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42153.log 2023-10-20T10:40:22.742 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44772.log 2023-10-20T10:40:22.742 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43386.log.gz 2023-10-20T10:40:22.743 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52136.log 2023-10-20T10:40:22.743 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42153.log.gz 2023-10-20T10:40:22.743 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44772.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44772.log.gz 2023-10-20T10:40:22.744 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45422.log 2023-10-20T10:40:22.744 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44253.log 2023-10-20T10:40:22.744 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52136.log.gz 2023-10-20T10:40:22.744 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48060.log 2023-10-20T10:40:22.745 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45422.log.gz 2023-10-20T10:40:22.745 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44741.log 2023-10-20T10:40:22.745 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44253.log.gz 2023-10-20T10:40:22.746 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44357.log 2023-10-20T10:40:22.746 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48060.log.gz 2023-10-20T10:40:22.746 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49450.log 2023-10-20T10:40:22.746 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44741.log.gz 2023-10-20T10:40:22.747 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40095.log 2023-10-20T10:40:22.747 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44357.log.gz 2023-10-20T10:40:22.747 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55086.log 2023-10-20T10:40:22.747 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49450.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49450.log.gz 2023-10-20T10:40:22.748 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44557.log 2023-10-20T10:40:22.748 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40095.log.gz 2023-10-20T10:40:22.748 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49237.log 2023-10-20T10:40:22.748 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55086.log.gz 2023-10-20T10:40:22.749 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.44557.log: /var/log/ceph/ceph-client.admin.47050.log 2023-10-20T10:40:22.749 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44557.log.gz 2023-10-20T10:40:22.749 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49974.log 2023-10-20T10:40:22.750 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49237.log.gz 2023-10-20T10:40:22.750 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40837.log 2023-10-20T10:40:22.750 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47050.log.gz 2023-10-20T10:40:22.750 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40260.log 2023-10-20T10:40:22.751 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49974.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49974.log.gz 2023-10-20T10:40:22.751 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50834.log 2023-10-20T10:40:22.751 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40837.log.gz 2023-10-20T10:40:22.752 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54305.log 2023-10-20T10:40:22.752 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40260.log.gz 2023-10-20T10:40:22.752 INFO:teuthology.orchestra.run.smithi150.stderr:gzip/var/log/ceph/ceph-client.admin.50834.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48157.log 2023-10-20T10:40:22.752 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50834.log.gz 2023-10-20T10:40:22.753 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47478.log 2023-10-20T10:40:22.753 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54305.log.gz 2023-10-20T10:40:22.753 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54794.log 2023-10-20T10:40:22.753 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48157.log.gz 2023-10-20T10:40:22.754 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54156.log 2023-10-20T10:40:22.754 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47478.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47478.log.gz 2023-10-20T10:40:22.754 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40962.log 2023-10-20T10:40:22.755 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54794.log.gz 2023-10-20T10:40:22.755 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40621.log 2023-10-20T10:40:22.755 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54156.log.gz 2023-10-20T10:40:22.755 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54379.log 2023-10-20T10:40:22.756 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40962.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40962.log.gz 2023-10-20T10:40:22.756 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56501.log 2023-10-20T10:40:22.756 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40621.log.gz 2023-10-20T10:40:22.756 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40314.log 2023-10-20T10:40:22.757 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54379.log.gz 2023-10-20T10:40:22.757 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51106.log 2023-10-20T10:40:22.757 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56501.log.gz 2023-10-20T10:40:22.758 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46187.log 2023-10-20T10:40:22.758 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40314.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40314.log.gz 2023-10-20T10:40:22.758 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52869.log 2023-10-20T10:40:22.758 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51106.log.gz 2023-10-20T10:40:22.759 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46187.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46187.log.gz 2023-10-20T10:40:22.759 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52869.log: 26.9% -- replaced with /var/log/ceph/ceph-client.admin.52869.log.gz 2023-10-20T10:40:22.760 INFO:teuthology.orchestra.run.smithi150.stderr: 90.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-20T10:40:23.027 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-10-20T10:40:38.236 INFO:teuthology.orchestra.run.smithi150.stderr: 89.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-20T10:40:41.783 INFO:teuthology.orchestra.run.smithi150.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-20T11:15:04.766 INFO:teuthology.orchestra.run.smithi150.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-20T11:25:44.605 INFO:teuthology.orchestra.run.smithi150.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-20T11:25:50.151 INFO:teuthology.orchestra.run.smithi150.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-20T11:25:50.152 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-20T11:25:50.152 INFO:teuthology.orchestra.run.smithi150.stderr:real 45m27.970s 2023-10-20T11:25:50.153 INFO:teuthology.orchestra.run.smithi150.stderr:user 41m51.940s 2023-10-20T11:25:50.153 INFO:teuthology.orchestra.run.smithi150.stderr:sys 2m11.827s 2023-10-20T11:25:50.154 INFO:tasks.ceph:Archiving logs... 2023-10-20T11:25:50.155 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/log/ceph to /home/teuthworker/archive/rraja-2023-10-20_08:52:26-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7432983/remote/smithi150/log 2023-10-20T11:25:50.156 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-20T11:27:47.160 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-20T11:27:47.175 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-20T11:27:47.176 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-20T11:27:47.230 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-20T11:27:47.242 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-20T11:27:47.243 DEBUG:teuthology.orchestra.run.smithi150:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-20T11:27:47.328 INFO:teuthology.orchestra.run.smithi150.stdout: remote refid st t when poll reach delay offset jitter 2023-10-20T11:27:47.328 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================== 2023-10-20T11:27:47.329 INFO:teuthology.orchestra.run.smithi150.stdout:*hv01.front.sepi 44.4.53.2 2 u 310 512 377 0.191 -6.440 1.055 2023-10-20T11:27:47.329 INFO:teuthology.orchestra.run.smithi150.stdout:+hv02.front.sepi 63.231.80.2 3 u 336 512 377 0.160 -3.960 0.850 2023-10-20T11:27:47.329 INFO:teuthology.orchestra.run.smithi150.stdout:+hv03.front.sepi 74.6.168.72 3 u 298 512 377 0.197 -3.863 1.485 2023-10-20T11:27:47.330 INFO:teuthology.orchestra.run.smithi150.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-20T11:27:47.331 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-20T11:27:47.342 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-20T11:27:47.343 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-20T11:27:47.377 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-20T11:27:47.411 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-20T11:27:47.442 INFO:teuthology.task.internal:Duration was 8664.112358 seconds 2023-10-20T11:27:47.442 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-20T11:27:47.478 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-20T11:27:47.478 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-20T11:27:47.821 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-20T11:27:47.822 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi150.front.sepia.ceph.com 2023-10-20T11:27:47.822 DEBUG:teuthology.orchestra.run.smithi150:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-20T11:27:47.882 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-20T11:27:47.882 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-20T11:27:47.969 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-20T11:27:47.970 DEBUG:teuthology.orchestra.run.smithi150:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-20T11:27:48.358 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-20T11:27:48.369 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-20T11:27:48.369 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-20T11:27:48.446 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-20T11:27:48.458 DEBUG:teuthology.orchestra.run.smithi150:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-20T11:27:48.479 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = core 2023-10-20T11:27:48.503 DEBUG:teuthology.orchestra.run.smithi150:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-20T11:27:48.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T11:27:48.552 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-20T11:27:48.565 INFO:teuthology.task.internal:Transferring archived files... 2023-10-20T11:27:48.567 DEBUG:teuthology.misc:Transferring archived files from smithi150:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rraja-2023-10-20_08:52:26-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7432983/remote/smithi150 2023-10-20T11:27:48.567 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-20T11:27:48.616 INFO:teuthology.task.internal:Removing archive directory... 2023-10-20T11:27:48.616 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-20T11:27:48.660 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-20T11:27:48.674 INFO:teuthology.task.internal:Not uploading archives. 2023-10-20T11:27:48.674 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-20T11:27:48.703 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-20T11:27:48.704 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-20T11:27:48.724 INFO:teuthology.orchestra.run.smithi150.stdout: 655472 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 20 11:27 /home/ubuntu/cephtest 2023-10-20T11:27:48.725 INFO:teuthology.orchestra.run.smithi150.stdout: 673333 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 20 09:13 /home/ubuntu/cephtest/mnt.0 2023-10-20T11:27:48.725 INFO:teuthology.orchestra.run.smithi150.stdout: 673759 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 20 09:15 /home/ubuntu/cephtest/mnt.0/client.0 2023-10-20T11:27:48.725 INFO:teuthology.orchestra.run.smithi150.stdout: 687666 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Oct 20 09:15 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-20T11:27:48.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T11:27:48.727 INFO:teuthology.orchestra.run.smithi150.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-10-20T11:27:48.727 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi150 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-20T11:27:48.735 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-20T11:27:48.748 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-20T11:27:48.779 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi150 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-20T11:27:48.780 INFO:teuthology.nuke:Checking targets against current locks 2023-10-20T11:27:48.823 DEBUG:teuthology.nuke:shortname: smithi150 2023-10-20T11:27:48.823 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-20T11:27:48.849 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rraja-2023-10-20_08:52:26-rbd:cli-ajarr-wip-extra-debug-62994-oct-19-distro-default-smithi/7432983', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-20 08:54:30.421475', 'locked_by': 'scheduled_rraja@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMhNZKSaorwOILbIWwC8yum4ac4p9BDJPPbLNv44G7t2YNONzis8NtcZXgBhQBBlI9Fi9CxItahxOgAuWR2Gut0='} 2023-10-20T11:27:48.897 INFO:teuthology.orchestra.console.smithi150:Power off 2023-10-20T11:27:48.898 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-20T11:27:48.923 DEBUG:teuthology.orchestra.console.smithi150:power off output: Chassis Power Control: Down/Off 2023-10-20T11:27:48.924 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-20T11:27:48.946 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2023-10-20T11:27:52.948 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-20T11:27:53.058 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2023-10-20T11:27:57.062 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-20T11:27:57.175 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2023-10-20T11:28:01.180 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-20T11:28:01.290 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off 2023-10-20T11:28:01.391 INFO:teuthology.orchestra.console.smithi150:Power off completed 2023-10-20T11:28:01.497 INFO:teuthology.run:Summary data: description: rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} duration: 8664.112358093262 failure_reason: 'Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi150 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7f103c7cad2e64eeb2f3003665c1f2f43a9381ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh''' flavor: default owner: scheduled_rraja@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=6223904771914abea1419e2dce9f3d24 status: fail success: false 2023-10-20T11:28:01.498 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-20T11:28:01.571 INFO:teuthology.run:FAIL